[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:43:08 dist-test-slave-wt2j krb5kdc[10178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:43:08 dist-test-slave-wt2j krb5kdc[10178](info): set up 2 sockets
Sep 26 16:43:08 dist-test-slave-wt2j krb5kdc[10178](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 16:43:09.750764 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.005s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:43:09 dist-test-slave-wt2j krb5kdc[10178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:43:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:43:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:43:09.957548 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:44777
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:37069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:44777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:10.293181 10191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:10.293581 10191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:10.293736 10191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:10.301128 10191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:43:10.301311 10191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:10.301402 10191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:10.301496 10191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:43:10.301569 10191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:43:10.311846 10191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:37069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:44777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:44777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:10.313396 10191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:10.314798 10191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:10.328850 10199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:10.328974 10196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:10.328884 10197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:10.329877 10191 server_base.cc:1034] running on GCE node
I20260926 16:43:10.330783 10191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:10.332564 10191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:10.333859 10191 hybrid_clock.cc:648] HybridClock initialized: now 1790440990333800 us; error 60 us; skew 500 ppm
I20260926 16:43:10.334378 10191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:10 dist-test-slave-wt2j krb5kdc[10178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440990, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:10.338200 10191 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:43:10.340126 10191 webserver.cc:460] Webserver started at http://127.9.238.254:44101/ using document root <none> and password file <none>
I20260926 16:43:10.340855 10191 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:10.340961 10191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:10.341212 10191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:10.344421 10191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "ae5df845c96a4f6b8bc3236302e17566"
format_stamp: "Formatted at 2026-09-26 16:43:10 on dist-test-slave-wt2j"
server_key: "f4efeb2b516acf7bd6bb52fa23a080a4"
server_key_iv: "8142d9acc0c52a305c4863de9e87ccd4"
server_key_version: "encryptionkey@0"
I20260926 16:43:10.345214 10191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "ae5df845c96a4f6b8bc3236302e17566"
format_stamp: "Formatted at 2026-09-26 16:43:10 on dist-test-slave-wt2j"
server_key: "f4efeb2b516acf7bd6bb52fa23a080a4"
server_key_iv: "8142d9acc0c52a305c4863de9e87ccd4"
server_key_version: "encryptionkey@0"
I20260926 16:43:10.351084 10191 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 16:43:10.355239 10206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:10.357084 10191 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:43:10.357270 10191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "ae5df845c96a4f6b8bc3236302e17566"
format_stamp: "Formatted at 2026-09-26 16:43:10 on dist-test-slave-wt2j"
server_key: "f4efeb2b516acf7bd6bb52fa23a080a4"
server_key_iv: "8142d9acc0c52a305c4863de9e87ccd4"
server_key_version: "encryptionkey@0"
I20260926 16:43:10.357446 10191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:10.374393 10191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:10.379055 10191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:10.379305 10191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:10.400477 10191 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:44777
I20260926 16:43:10.400629 10257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:44777 every 8 connection(s)
I20260926 16:43:10.402004 10191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:43:10.407680 10258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:10.410441 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 10191
I20260926 16:43:10.410888 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:43:10.414036 10171 external_mini_cluster.cc:1428] Setting key dec5c1017b40e551fc9178d0098aaa8e
I20260926 16:43:10.425657 10258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae5df845c96a4f6b8bc3236302e17566: Bootstrap starting.
I20260926 16:43:10.429697 10258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae5df845c96a4f6b8bc3236302e17566: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:10.431602 10258 log.cc:824] T 00000000000000000000000000000000 P ae5df845c96a4f6b8bc3236302e17566: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:10.436162 10258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae5df845c96a4f6b8bc3236302e17566: No bootstrap required, opened a new log
I20260926 16:43:10.447431 10258 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae5df845c96a4f6b8bc3236302e17566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5df845c96a4f6b8bc3236302e17566" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44777 } }
I20260926 16:43:10.448069 10258 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae5df845c96a4f6b8bc3236302e17566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:10.448225 10258 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae5df845c96a4f6b8bc3236302e17566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5df845c96a4f6b8bc3236302e17566, State: Initialized, Role: FOLLOWER
I20260926 16:43:10.449232 10258 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae5df845c96a4f6b8bc3236302e17566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5df845c96a4f6b8bc3236302e17566" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44777 } }
I20260926 16:43:10.449661 10258 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae5df845c96a4f6b8bc3236302e17566 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:43:10.449852 10258 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae5df845c96a4f6b8bc3236302e17566 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:43:10.450109 10258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae5df845c96a4f6b8bc3236302e17566 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 16:43:10 dist-test-slave-wt2j krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:10.453554 10258 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae5df845c96a4f6b8bc3236302e17566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5df845c96a4f6b8bc3236302e17566" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44777 } }
I20260926 16:43:10.454574 10258 leader_election.cc:302] T 00000000000000000000000000000000 P ae5df845c96a4f6b8bc3236302e17566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae5df845c96a4f6b8bc3236302e17566; no voters:
I20260926 16:43:10.455618 10258 leader_election.cc:288] T 00000000000000000000000000000000 P ae5df845c96a4f6b8bc3236302e17566 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:43:10.455866 10263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae5df845c96a4f6b8bc3236302e17566 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:10.456815 10263 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae5df845c96a4f6b8bc3236302e17566 [term 1 LEADER]: Becoming Leader. State: Replica: ae5df845c96a4f6b8bc3236302e17566, State: Running, Role: LEADER
I20260926 16:43:10.457589 10263 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae5df845c96a4f6b8bc3236302e17566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5df845c96a4f6b8bc3236302e17566" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44777 } }
I20260926 16:43:10.460701 10258 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae5df845c96a4f6b8bc3236302e17566 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:43:10.462472 10265 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae5df845c96a4f6b8bc3236302e17566 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae5df845c96a4f6b8bc3236302e17566. Latest consensus state: current_term: 1 leader_uuid: "ae5df845c96a4f6b8bc3236302e17566" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5df845c96a4f6b8bc3236302e17566" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44777 } } }
I20260926 16:43:10.463014 10265 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae5df845c96a4f6b8bc3236302e17566 [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:10.464584 10268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:43:10.465808 10261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:10.420560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55376 (local address 127.9.238.254:44777)
0926 16:43:10.423301 (+ 2741us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:10.423318 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:10.423383 (+ 65us) server_negotiation.cc:407] Connection header received
0926 16:43:10.433481 (+ 10098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:10.433523 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:10.434222 (+ 699us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:10.434862 (+ 640us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:10.438398 (+ 3536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.441274 (+ 2876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.443318 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:10.444723 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:10.445032 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:10.452260 (+ 7228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:10.452280 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:10.452315 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:10.452401 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:10.458132 (+ 5731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:10.459270 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:10.459280 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:10.459287 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:10.459627 (+ 340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:10.460153 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:10.460161 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:10.460166 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:10.461426 (+ 1260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:10.461835 (+ 409us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:10.462872 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0926 16:43:10.463411 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2421,"thread_start_us":2052,"threads_started":1}
I20260926 16:43:10.463894 10264 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae5df845c96a4f6b8bc3236302e17566 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae5df845c96a4f6b8bc3236302e17566" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5df845c96a4f6b8bc3236302e17566" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44777 } } }
I20260926 16:43:10.466799 10264 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae5df845c96a4f6b8bc3236302e17566 [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:10.476032 10268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:43:10.495397 10268 catalog_manager.cc:1347] Generated new cluster ID: 03433d5abb4a401c90c620ed3de189f8
I20260926 16:43:10.495647 10268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:43:10.511816 10268 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:43:10.513243 10268 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:43:10.523720 10268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae5df845c96a4f6b8bc3236302e17566: Generated new TSK 0
I20260926 16:43:10.525221 10268 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
I20260926 16:43:10.585796 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:0
--local_ip_for_outbound_sockets=127.9.238.193
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44777
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:37069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:10.939394 10284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:10.939838 10284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:10.940033 10284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:10.948318 10284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:10.948514 10284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:10.948731 10284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:43:10.960579 10284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:37069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44777
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:10.962653 10284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:10.964540 10284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:10.980625 10290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:10.980684 10292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:10.980625 10289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:10.981681 10284 server_base.cc:1034] running on GCE node
I20260926 16:43:10.982475 10284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:10.983815 10284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:10.985087 10284 hybrid_clock.cc:648] HybridClock initialized: now 1790440990985057 us; error 83 us; skew 500 ppm
I20260926 16:43:10.985566 10284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:10 dist-test-slave-wt2j krb5kdc[10178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440990, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:10.989594 10284 init.cc:377] Logged in from keytab as kudu/127.9.238.193@KRBTEST.COM (short username kudu)
I20260926 16:43:10.991493 10284 webserver.cc:460] Webserver started at http://127.9.238.193:37431/ using document root <none> and password file <none>
I20260926 16:43:10.992441 10284 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:10.992563 10284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:10.992889 10284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:10.996711 10284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-0/data/instance:
uuid: "b731abc58da14e1ab15b78906aa926ff"
format_stamp: "Formatted at 2026-09-26 16:43:10 on dist-test-slave-wt2j"
server_key: "be0c135d1d4f37eb85d4bd3eebbb8a26"
server_key_iv: "63786cec6a3b1cfb25245cf628d48d47"
server_key_version: "encryptionkey@0"
I20260926 16:43:10.997699 10284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance:
uuid: "b731abc58da14e1ab15b78906aa926ff"
format_stamp: "Formatted at 2026-09-26 16:43:10 on dist-test-slave-wt2j"
server_key: "be0c135d1d4f37eb85d4bd3eebbb8a26"
server_key_iv: "63786cec6a3b1cfb25245cf628d48d47"
server_key_version: "encryptionkey@0"
I20260926 16:43:11.005102 10284 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260926 16:43:11.009840 10299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:11.012012 10284 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 16:43:11.012217 10284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "b731abc58da14e1ab15b78906aa926ff"
format_stamp: "Formatted at 2026-09-26 16:43:10 on dist-test-slave-wt2j"
server_key: "be0c135d1d4f37eb85d4bd3eebbb8a26"
server_key_iv: "63786cec6a3b1cfb25245cf628d48d47"
server_key_version: "encryptionkey@0"
I20260926 16:43:11.012459 10284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:11.028748 10284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:11.034078 10284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:11.034394 10284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:11.035663 10284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:11.037348 10284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:11.037431 10284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:11.037559 10284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:11.037633 10284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:11.083962 10284 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:35661
I20260926 16:43:11.084080 10411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:35661 every 8 connection(s)
I20260926 16:43:11.085711 10284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:43:11.089162 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 10284
I20260926 16:43:11.089597 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance
I20260926 16:43:11.090597 10171 external_mini_cluster.cc:1428] Setting key 9426397737651dc1affe9714c191a00c
Sep 26 16:43:11 dist-test-slave-wt2j krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440990, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.238.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.194@KRBTEST.COM" created.
I20260926 16:43:11.118453 10414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:11.089907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:37507 (local address 127.9.238.254:44777)
0926 16:43:11.090423 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:11.090429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:11.093987 (+ 3558us) server_negotiation.cc:407] Connection header received
0926 16:43:11.096647 (+ 2660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:11.096656 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:11.096772 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:11.096956 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:11.099880 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:11.100966 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:11.104257 (+ 3291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:11.104745 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:11.104885 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:11.110538 (+ 5653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:11.110551 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:11.110558 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:11.110623 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:11.114656 (+ 4033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:11.115758 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:11.115763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:11.115766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:11.115938 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:11.116423 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:11.116436 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:11.116438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:11.116733 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:11.116953 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:11.117784 (+ 831us) server_negotiation.cc:299] Negotiation successful
0926 16:43:11.118040 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":284,"threads_started":1}
I20260926 16:43:11.119840 10415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:11.091831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:44777 (local address 127.9.238.193:37507)
0926 16:43:11.093475 (+ 1644us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:11.093550 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:11.096185 (+ 2635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:11.097382 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:11.097402 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:11.098510 (+ 1108us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:11.099612 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:11.099651 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:11.101176 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:11.101186 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:11.104018 (+ 2832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:11.104042 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:11.105084 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:11.105094 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:11.105489 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:11.106525 (+ 1036us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:11.106565 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:11.110118 (+ 3553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:11.114916 (+ 4798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:11.114926 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:11.114942 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:11.115517 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:11.116079 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:11.116084 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:11.116092 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:11.116291 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:11.116918 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:11.116928 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:11.117376 (+ 448us) client_negotiation.cc:769] Sending connection context
0926 16:43:11.117683 (+ 307us) client_negotiation.cc:240] Negotiation successful
0926 16:43:11.118277 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1084,"thread_start_us":719,"threads_started":1}
I20260926 16:43:11.121707 10412 heartbeater.cc:347] Connected to a master server at 127.9.238.254:44777
I20260926 16:43:11.122195 10412 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:11.123384 10412 heartbeater.cc:511] Master 127.9.238.254:44777 requested a full tablet report, sending...
I20260926 16:43:11.126346 10223 ts_manager.cc:194] Registered new tserver with Master: b731abc58da14e1ab15b78906aa926ff (127.9.238.193:35661)
I20260926 16:43:11.128183 10223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.193@KRBTEST.COM'} at 127.9.238.193:37507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
I20260926 16:43:11.147819 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:0
--local_ip_for_outbound_sockets=127.9.238.194
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44777
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:37069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:11.554152 10418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:11.554553 10418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:11.554762 10418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:11.563669 10418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:11.563840 10418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:11.564081 10418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:43:11.575539 10418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:37069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44777
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:11.577435 10418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:11.579293 10418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:11.596769 10426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:11.596710 10424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:11.596817 10423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:11.598573 10418 server_base.cc:1034] running on GCE node
I20260926 16:43:11.599714 10418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:11.601048 10418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:11.602370 10418 hybrid_clock.cc:648] HybridClock initialized: now 1790440991602309 us; error 40 us; skew 500 ppm
I20260926 16:43:11.602790 10418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:11 dist-test-slave-wt2j krb5kdc[10178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440991, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:11.607038 10418 init.cc:377] Logged in from keytab as kudu/127.9.238.194@KRBTEST.COM (short username kudu)
I20260926 16:43:11.609551 10418 webserver.cc:460] Webserver started at http://127.9.238.194:37985/ using document root <none> and password file <none>
I20260926 16:43:11.610647 10418 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:11.610829 10418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:11.611268 10418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:11.615159 10418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-1/data/instance:
uuid: "25f6b6a9c2ca4dbba3f130c6ac6d4f8f"
format_stamp: "Formatted at 2026-09-26 16:43:11 on dist-test-slave-wt2j"
server_key: "05ce899503459708f8cccd9e3a63e3e8"
server_key_iv: "f678d72f54792ac9b1a13ce23159c839"
server_key_version: "encryptionkey@0"
I20260926 16:43:11.616245 10418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance:
uuid: "25f6b6a9c2ca4dbba3f130c6ac6d4f8f"
format_stamp: "Formatted at 2026-09-26 16:43:11 on dist-test-slave-wt2j"
server_key: "05ce899503459708f8cccd9e3a63e3e8"
server_key_iv: "f678d72f54792ac9b1a13ce23159c839"
server_key_version: "encryptionkey@0"
I20260926 16:43:11.623773 10418 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260926 16:43:11.629182 10433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:11.631390 10418 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260926 16:43:11.631713 10418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "25f6b6a9c2ca4dbba3f130c6ac6d4f8f"
format_stamp: "Formatted at 2026-09-26 16:43:11 on dist-test-slave-wt2j"
server_key: "05ce899503459708f8cccd9e3a63e3e8"
server_key_iv: "f678d72f54792ac9b1a13ce23159c839"
server_key_version: "encryptionkey@0"
I20260926 16:43:11.631983 10418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:11.657718 10418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:11.663869 10418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:11.664263 10418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:11.665567 10418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:11.667224 10418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:11.667305 10418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:11.667469 10418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:11.667524 10418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:11.707783 10418 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:44283
I20260926 16:43:11.707895 10545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:44283 every 8 connection(s)
I20260926 16:43:11.709468 10418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:43:11.710192 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 10418
I20260926 16:43:11.710445 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance
I20260926 16:43:11.711038 10171 external_mini_cluster.cc:1428] Setting key 2fe4a3bf296fbd22d2e6e7b41049c9c2
Sep 26 16:43:11 dist-test-slave-wt2j krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440991, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.238.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.195@KRBTEST.COM" created.
I20260926 16:43:11.736649 10548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:11.712911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:60657 (local address 127.9.238.254:44777)
0926 16:43:11.713487 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:11.713502 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:11.714981 (+ 1479us) server_negotiation.cc:407] Connection header received
0926 16:43:11.717377 (+ 2396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:11.717385 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:11.717499 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:11.717667 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:11.720015 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:11.721323 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:11.723288 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:11.724129 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:11.724292 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:11.728904 (+ 4612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:11.728912 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:11.728920 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:11.729017 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:11.732183 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:11.733387 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:11.733391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:11.733394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:11.733557 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:11.734216 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:11.734220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:11.734222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:11.734538 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:11.734693 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:11.736084 (+ 1391us) server_negotiation.cc:299] Negotiation successful
0926 16:43:11.736295 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":295,"threads_started":1}
I20260926 16:43:11.738554 10549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:11.713749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:44777 (local address 127.9.238.194:60657)
0926 16:43:11.714756 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:11.714816 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:11.716941 (+ 2125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:11.718033 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:11.718044 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:11.718915 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:11.719765 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:11.719790 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:11.721534 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:11.721539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:11.723097 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:11.723113 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:11.724421 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:11.724427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:11.724701 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:11.725616 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:11.725646 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:11.728589 (+ 2943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:11.732432 (+ 3843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:11.732442 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:11.732462 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:11.733207 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:11.733697 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:11.733705 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:11.733712 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:11.734029 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:11.734703 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:11.734716 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:11.735412 (+ 696us) client_negotiation.cc:769] Sending connection context
0926 16:43:11.735852 (+ 440us) client_negotiation.cc:240] Negotiation successful
0926 16:43:11.736620 (+ 768us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":396,"threads_started":1}
I20260926 16:43:11.740585 10546 heartbeater.cc:347] Connected to a master server at 127.9.238.254:44777
I20260926 16:43:11.741082 10546 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:11.742206 10546 heartbeater.cc:511] Master 127.9.238.254:44777 requested a full tablet report, sending...
I20260926 16:43:11.744287 10223 ts_manager.cc:194] Registered new tserver with Master: 25f6b6a9c2ca4dbba3f130c6ac6d4f8f (127.9.238.194:44283)
I20260926 16:43:11.745277 10223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.194@KRBTEST.COM'} at 127.9.238.194:60657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
I20260926 16:43:11.760782 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:0
--local_ip_for_outbound_sockets=127.9.238.195
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44777
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:37069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:12.124864 10552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:12.125236 10552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:12.125407 10552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 16:43:12.131611 10412 heartbeater.cc:503] Master 127.9.238.254:44777 was elected leader, sending a full tablet report...
W20260926 16:43:12.133641 10552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:12.133754 10552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:12.133942 10552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:43:12.144852 10552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:37069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44777
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:12.146323 10552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:12.147787 10552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:12.162599 10557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:12.162638 10558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:12.163010 10560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:12.163725 10552 server_base.cc:1034] running on GCE node
I20260926 16:43:12.164384 10552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:12.165514 10552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:12.166779 10552 hybrid_clock.cc:648] HybridClock initialized: now 1790440992166722 us; error 64 us; skew 500 ppm
I20260926 16:43:12.167176 10552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:12 dist-test-slave-wt2j krb5kdc[10178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440992, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:12.170806 10552 init.cc:377] Logged in from keytab as kudu/127.9.238.195@KRBTEST.COM (short username kudu)
I20260926 16:43:12.172593 10552 webserver.cc:460] Webserver started at http://127.9.238.195:37177/ using document root <none> and password file <none>
I20260926 16:43:12.173305 10552 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:12.173442 10552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:12.173761 10552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:12.176891 10552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-2/data/instance:
uuid: "f7b3139a60c14f69b18e32bb9e9912cc"
format_stamp: "Formatted at 2026-09-26 16:43:12 on dist-test-slave-wt2j"
server_key: "476321a9ebe1e48c653f60c6048e8cf3"
server_key_iv: "ed21d3bc38e46c63ea1a72942ab4c9e5"
server_key_version: "encryptionkey@0"
I20260926 16:43:12.177680 10552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance:
uuid: "f7b3139a60c14f69b18e32bb9e9912cc"
format_stamp: "Formatted at 2026-09-26 16:43:12 on dist-test-slave-wt2j"
server_key: "476321a9ebe1e48c653f60c6048e8cf3"
server_key_iv: "ed21d3bc38e46c63ea1a72942ab4c9e5"
server_key_version: "encryptionkey@0"
I20260926 16:43:12.183753 10552 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 16:43:12.188081 10567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:12.189865 10552 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:43:12.190245 10552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "f7b3139a60c14f69b18e32bb9e9912cc"
format_stamp: "Formatted at 2026-09-26 16:43:12 on dist-test-slave-wt2j"
server_key: "476321a9ebe1e48c653f60c6048e8cf3"
server_key_iv: "ed21d3bc38e46c63ea1a72942ab4c9e5"
server_key_version: "encryptionkey@0"
I20260926 16:43:12.190519 10552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:12.211910 10552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:12.216931 10552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:12.217203 10552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:12.218286 10552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:12.219774 10552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:12.219854 10552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:12.219945 10552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:12.219980 10552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:12.260215 10552 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:33513
I20260926 16:43:12.260354 10679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:33513 every 8 connection(s)
I20260926 16:43:12.261802 10552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:43:12.266084 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 10552
I20260926 16:43:12.266346 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance
I20260926 16:43:12.266842 10171 external_mini_cluster.cc:1428] Setting key 6d490b83c1cbcea64f154aec2ea4a6d9
Sep 26 16:43:12 dist-test-slave-wt2j krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440992, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:12.285653 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:12.265299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:34081 (local address 127.9.238.254:44777)
0926 16:43:12.265888 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:12.265896 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:12.267016 (+ 1120us) server_negotiation.cc:407] Connection header received
0926 16:43:12.269541 (+ 2525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:12.269554 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:12.269642 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:12.269802 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:12.271924 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:12.272852 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:12.274498 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:12.274943 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:12.275055 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:12.278665 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:12.278670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:12.278674 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:12.278748 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:12.281923 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:12.282936 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:12.282941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:12.282952 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:12.283166 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:12.283718 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:12.283723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:12.283726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:12.284070 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:12.284270 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:12.284965 (+ 695us) server_negotiation.cc:299] Negotiation successful
0926 16:43:12.285221 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":306,"threads_started":1}
I20260926 16:43:12.286695 10683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:12.265949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:44777 (local address 127.9.238.195:34081)
0926 16:43:12.266849 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:12.266897 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:12.269227 (+ 2330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:12.270093 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:12.270108 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:12.270895 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:12.271722 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:12.271743 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:12.273045 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:12.273049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:12.274332 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:12.274345 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:12.275210 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:12.275216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:12.275424 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:12.276230 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:12.276257 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:12.278394 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:12.282109 (+ 3715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:12.282135 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:12.282149 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:12.282743 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:12.283299 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:12.283313 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:12.283315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:12.283504 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:12.284222 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:12.284231 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:12.284598 (+ 367us) client_negotiation.cc:769] Sending connection context
0926 16:43:12.284873 (+ 275us) client_negotiation.cc:240] Negotiation successful
0926 16:43:12.285446 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":379,"threads_started":1}
I20260926 16:43:12.288681 10680 heartbeater.cc:347] Connected to a master server at 127.9.238.254:44777
I20260926 16:43:12.289099 10680 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:12.289880 10680 heartbeater.cc:511] Master 127.9.238.254:44777 requested a full tablet report, sending...
I20260926 16:43:12.291966 10223 ts_manager.cc:194] Registered new tserver with Master: f7b3139a60c14f69b18e32bb9e9912cc (127.9.238.195:33513)
I20260926 16:43:12.293032 10223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.195@KRBTEST.COM'} at 127.9.238.195:34081
I20260926 16:43:12.297673 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 16:43:12 dist-test-slave-wt2j krb5kdc[10178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440992, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 16:43:12 dist-test-slave-wt2j krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440992, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.238.193@KRBTEST.COM
I20260926 16:43:12.350457 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:12.332194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50924 (local address 127.9.238.193:35661)
0926 16:43:12.333054 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:12.333063 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:12.333105 (+ 42us) server_negotiation.cc:407] Connection header received
0926 16:43:12.333199 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:12.333206 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:12.333440 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:12.333586 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:12.335652 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:12.336736 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:12.338408 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:12.338986 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:12.339150 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:12.343919 (+ 4769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:12.343930 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:12.343945 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:12.343987 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:12.347143 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:12.348133 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:12.348140 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:12.348143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:12.348265 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:12.348852 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:12.348856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:12.348858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:12.349242 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:12.349517 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:12.349830 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 16:43:12.350043 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":509,"threads_started":1}
W20260926 16:43:12.352490 10327 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50924
Sep 26 16:43:12 dist-test-slave-wt2j krb5kdc[10178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 16:43:12 dist-test-slave-wt2j krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.193@KRBTEST.COM
I20260926 16:43:12.405879 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:12.388872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50928 (local address 127.9.238.193:35661)
0926 16:43:12.389247 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:12.389254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:12.389778 (+ 524us) server_negotiation.cc:407] Connection header received
0926 16:43:12.389957 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:12.389961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:12.390093 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:12.390205 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:12.391999 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:12.392712 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:12.394118 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:12.394664 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:12.394802 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:12.399658 (+ 4856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:12.399664 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:12.399669 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:12.399714 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:12.402787 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:12.403887 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:12.403892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:12.403895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:12.404106 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:12.404575 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:12.404578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:12.404581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:12.404830 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:12.405015 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:12.405434 (+ 419us) server_negotiation.cc:299] Negotiation successful
0926 16:43:12.405634 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260926 16:43:12.408174 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 10284
I20260926 16:43:12.424047 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:12.409578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50944 (local address 127.9.238.193:35661)
0926 16:43:12.409795 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:12.409801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:12.410071 (+ 270us) server_negotiation.cc:407] Connection header received
0926 16:43:12.410204 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:12.410208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:12.410286 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:12.410517 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:12.411987 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:12.412757 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:12.414148 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:12.414603 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:12.414743 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:12.417526 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:12.417532 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:12.417537 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:12.417573 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:12.420824 (+ 3251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:12.421687 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:12.421695 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:12.421699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:12.421876 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:12.422299 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:12.422305 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:12.422308 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:12.422641 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:12.422954 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:12.423300 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 16:43:12.423619 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 16:43:12.425055 10407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:43:12.513738 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 10284
I20260926 16:43:12.534165 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 10418
Sep 26 16:43:12 dist-test-slave-wt2j krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.194@KRBTEST.COM
I20260926 16:43:12.554361 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:12.535351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46834 (local address 127.9.238.194:44283)
0926 16:43:12.536087 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:12.536096 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:12.536130 (+ 34us) server_negotiation.cc:407] Connection header received
0926 16:43:12.536214 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:12.536222 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:12.536527 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:12.536818 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:12.538361 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:12.539726 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:12.541083 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:12.541735 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:12.541894 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:12.547066 (+ 5172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:12.547083 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:12.547104 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:12.547164 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:12.550923 (+ 3759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:12.551981 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:12.551990 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:12.551993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:12.552167 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:12.552680 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:12.552683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:12.552686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:12.553143 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:12.553446 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:12.553721 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 16:43:12.554012 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":389,"threads_started":1}
I20260926 16:43:12.556397 10541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:43:12.643460 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 10418
I20260926 16:43:12.669533 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 10552
Sep 26 16:43:12 dist-test-slave-wt2j krb5kdc[10178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.195@KRBTEST.COM
I20260926 16:43:12.688037 10697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:12.670891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50098 (local address 127.9.238.195:33513)
0926 16:43:12.671809 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:12.671818 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:12.671853 (+ 35us) server_negotiation.cc:407] Connection header received
0926 16:43:12.671920 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:12.671928 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:12.672193 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:12.672386 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:12.673665 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:12.674731 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:12.676106 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:12.676717 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:12.676873 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:12.681471 (+ 4598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:12.681484 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:12.681501 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:12.681543 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:12.684905 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:12.685685 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:12.685694 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:12.685696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:12.685828 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:12.686217 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:12.686222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:12.686228 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:12.686610 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:12.686965 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:12.687285 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 16:43:12.687680 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":422,"threads_started":1}
I20260926 16:43:12.690002 10675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:43:12.783885 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 10552
I20260926 16:43:12.803396 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 10191
I20260926 16:43:12.804958 10253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:43:12.924322 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 10191
2026-09-26T16:43:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4316 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:43:13 dist-test-slave-wt2j krb5kdc[10702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:43:13 dist-test-slave-wt2j krb5kdc[10702](info): set up 2 sockets
Sep 26 16:43:13 dist-test-slave-wt2j krb5kdc[10702](info): commencing operation
krb5kdc: starting...
W20260926 16:43:14.064854 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:43:14 dist-test-slave-wt2j krb5kdc[10702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:43:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:43:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:43:14.276315 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:39693
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:46567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:39693
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:14.692981 10715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:14.693430 10715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:14.693603 10715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:14.701967 10715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:43:14.702163 10715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:14.702277 10715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:14.702394 10715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:43:14.702476 10715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:43:14.714285 10715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:46567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:39693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:39693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:14.716391 10715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:14.718237 10715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:14.733587 10720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:14.733529 10721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:14.733525 10723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:14.734730 10715 server_base.cc:1034] running on GCE node
I20260926 16:43:14.735864 10715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:14.737730 10715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:14.739156 10715 hybrid_clock.cc:648] HybridClock initialized: now 1790440994739042 us; error 120 us; skew 500 ppm
I20260926 16:43:14.739732 10715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:14 dist-test-slave-wt2j krb5kdc[10702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440994, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:14.743441 10715 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:43:14.745374 10715 webserver.cc:460] Webserver started at http://127.9.238.254:35501/ using document root <none> and password file <none>
I20260926 16:43:14.746515 10715 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:14.746654 10715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:14.747006 10715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:14.750715 10715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "44d055d92e6643e9badfb5fdc86f36e3"
format_stamp: "Formatted at 2026-09-26 16:43:14 on dist-test-slave-wt2j"
server_key: "24997bda26bed450427eb4cd6ea3b461"
server_key_iv: "5715ddcdc8d09731eb3fb268b47c03fd"
server_key_version: "encryptionkey@0"
I20260926 16:43:14.751533 10715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "44d055d92e6643e9badfb5fdc86f36e3"
format_stamp: "Formatted at 2026-09-26 16:43:14 on dist-test-slave-wt2j"
server_key: "24997bda26bed450427eb4cd6ea3b461"
server_key_iv: "5715ddcdc8d09731eb3fb268b47c03fd"
server_key_version: "encryptionkey@0"
I20260926 16:43:14.758092 10715 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 16:43:14.762442 10730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:14.764442 10715 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 16:43:14.764686 10715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "44d055d92e6643e9badfb5fdc86f36e3"
format_stamp: "Formatted at 2026-09-26 16:43:14 on dist-test-slave-wt2j"
server_key: "24997bda26bed450427eb4cd6ea3b461"
server_key_iv: "5715ddcdc8d09731eb3fb268b47c03fd"
server_key_version: "encryptionkey@0"
I20260926 16:43:14.764905 10715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:14.783735 10715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:14.788496 10715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:14.788793 10715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:14.810384 10715 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:39693
I20260926 16:43:14.810418 10781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:39693 every 8 connection(s)
I20260926 16:43:14.811983 10715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:43:14.817127 10782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:14.820384 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 10715
I20260926 16:43:14.820648 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:43:14.821280 10171 external_mini_cluster.cc:1428] Setting key 0eb351f00c94fe7a68549ee744899e4b
I20260926 16:43:14.832401 10782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44d055d92e6643e9badfb5fdc86f36e3: Bootstrap starting.
I20260926 16:43:14.836921 10782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44d055d92e6643e9badfb5fdc86f36e3: Neither blocks nor log segments found. Creating new log.
Sep 26 16:43:14 dist-test-slave-wt2j krb5kdc[10702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:14.838351 10782 log.cc:824] T 00000000000000000000000000000000 P 44d055d92e6643e9badfb5fdc86f36e3: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:14.842657 10782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44d055d92e6643e9badfb5fdc86f36e3: No bootstrap required, opened a new log
I20260926 16:43:14.847133 10785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:14.823863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39638 (local address 127.9.238.254:39693)
0926 16:43:14.825250 (+ 1387us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:14.825278 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:14.825357 (+ 79us) server_negotiation.cc:407] Connection header received
0926 16:43:14.827728 (+ 2371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:14.827761 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:14.828325 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:14.828829 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:14.830394 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:14.831952 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:14.833297 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:14.834030 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:14.834257 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:14.838511 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:14.838520 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:14.838572 (+ 52us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:14.838639 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:14.842450 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:14.843216 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:14.843223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:14.843231 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:14.843429 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:14.843838 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:14.843845 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:14.843848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:14.844474 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:14.844839 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:14.845412 (+ 573us) server_negotiation.cc:299] Negotiation successful
0926 16:43:14.845860 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":772,"thread_start_us":449,"threads_started":1}
I20260926 16:43:14.853099 10782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44d055d92e6643e9badfb5fdc86f36e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d055d92e6643e9badfb5fdc86f36e3" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 39693 } }
I20260926 16:43:14.853629 10782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44d055d92e6643e9badfb5fdc86f36e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:14.853749 10782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44d055d92e6643e9badfb5fdc86f36e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d055d92e6643e9badfb5fdc86f36e3, State: Initialized, Role: FOLLOWER
I20260926 16:43:14.854471 10782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44d055d92e6643e9badfb5fdc86f36e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d055d92e6643e9badfb5fdc86f36e3" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 39693 } }
I20260926 16:43:14.854794 10782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44d055d92e6643e9badfb5fdc86f36e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:43:14.854962 10782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44d055d92e6643e9badfb5fdc86f36e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:43:14.855160 10782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44d055d92e6643e9badfb5fdc86f36e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:14.858073 10782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44d055d92e6643e9badfb5fdc86f36e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d055d92e6643e9badfb5fdc86f36e3" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 39693 } }
I20260926 16:43:14.858726 10782 leader_election.cc:302] T 00000000000000000000000000000000 P 44d055d92e6643e9badfb5fdc86f36e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44d055d92e6643e9badfb5fdc86f36e3; no voters:
I20260926 16:43:14.859499 10782 leader_election.cc:288] T 00000000000000000000000000000000 P 44d055d92e6643e9badfb5fdc86f36e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:43:14.859745 10787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44d055d92e6643e9badfb5fdc86f36e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:14.860685 10787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44d055d92e6643e9badfb5fdc86f36e3 [term 1 LEADER]: Becoming Leader. State: Replica: 44d055d92e6643e9badfb5fdc86f36e3, State: Running, Role: LEADER
I20260926 16:43:14.861459 10787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44d055d92e6643e9badfb5fdc86f36e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d055d92e6643e9badfb5fdc86f36e3" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 39693 } }
I20260926 16:43:14.862494 10782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44d055d92e6643e9badfb5fdc86f36e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:43:14.865497 10789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44d055d92e6643e9badfb5fdc86f36e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44d055d92e6643e9badfb5fdc86f36e3. Latest consensus state: current_term: 1 leader_uuid: "44d055d92e6643e9badfb5fdc86f36e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d055d92e6643e9badfb5fdc86f36e3" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 39693 } } }
I20260926 16:43:14.866019 10789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44d055d92e6643e9badfb5fdc86f36e3 [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:14.866575 10788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44d055d92e6643e9badfb5fdc86f36e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44d055d92e6643e9badfb5fdc86f36e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44d055d92e6643e9badfb5fdc86f36e3" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 39693 } } }
I20260926 16:43:14.866946 10788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44d055d92e6643e9badfb5fdc86f36e3 [sys.catalog]: This master's current role is: LEADER
W20260926 16:43:14.878623 10803 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 44d055d92e6643e9badfb5fdc86f36e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:43:14.878813 10803 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:43:14.879017 10796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:43:14.880846 10796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:43:14.891219 10796 catalog_manager.cc:1347] Generated new cluster ID: df1cb7998e274e3fbec1098dfbf91a34
I20260926 16:43:14.891381 10796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:43:14.906297 10796 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:43:14.908059 10796 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:43:14.917112 10796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44d055d92e6643e9badfb5fdc86f36e3: Generated new TSK 0
I20260926 16:43:14.918092 10796 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
I20260926 16:43:14.967917 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:0
--local_ip_for_outbound_sockets=127.9.238.193
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:39693
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:46567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:15.385105 10808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:15.385480 10808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:15.385650 10808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:15.393818 10808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:15.393978 10808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:15.394191 10808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:43:15.405534 10808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:46567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:39693
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:15.407330 10808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:15.409148 10808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:15.425213 10813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:15.425244 10814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:15.425460 10816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:15.426254 10808 server_base.cc:1034] running on GCE node
I20260926 16:43:15.427024 10808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:15.428254 10808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:15.429545 10808 hybrid_clock.cc:648] HybridClock initialized: now 1790440995429461 us; error 77 us; skew 500 ppm
I20260926 16:43:15.429960 10808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:15 dist-test-slave-wt2j krb5kdc[10702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440995, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:15.433647 10808 init.cc:377] Logged in from keytab as kudu/127.9.238.193@KRBTEST.COM (short username kudu)
I20260926 16:43:15.435634 10808 webserver.cc:460] Webserver started at http://127.9.238.193:33753/ using document root <none> and password file <none>
I20260926 16:43:15.436467 10808 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:15.436620 10808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:15.436957 10808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:15.440317 10808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-0/data/instance:
uuid: "d6f6fb3f5bd84a3fb1060f27086dd946"
format_stamp: "Formatted at 2026-09-26 16:43:15 on dist-test-slave-wt2j"
server_key: "dd772015339ccd30e8d7be275353a710"
server_key_iv: "7526e8a42ddcd13ad57f26a87cf68fa7"
server_key_version: "encryptionkey@0"
I20260926 16:43:15.441164 10808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance:
uuid: "d6f6fb3f5bd84a3fb1060f27086dd946"
format_stamp: "Formatted at 2026-09-26 16:43:15 on dist-test-slave-wt2j"
server_key: "dd772015339ccd30e8d7be275353a710"
server_key_iv: "7526e8a42ddcd13ad57f26a87cf68fa7"
server_key_version: "encryptionkey@0"
I20260926 16:43:15.447285 10808 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260926 16:43:15.452335 10823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:15.454419 10808 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 16:43:15.454674 10808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "d6f6fb3f5bd84a3fb1060f27086dd946"
format_stamp: "Formatted at 2026-09-26 16:43:15 on dist-test-slave-wt2j"
server_key: "dd772015339ccd30e8d7be275353a710"
server_key_iv: "7526e8a42ddcd13ad57f26a87cf68fa7"
server_key_version: "encryptionkey@0"
I20260926 16:43:15.454911 10808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:15.507515 10808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:15.512341 10808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:15.512665 10808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:15.513780 10808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:15.515332 10808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:15.515442 10808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:15.515627 10808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:15.515703 10808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260926 16:43:15.555809 10808 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:41337
I20260926 16:43:15.555922 10935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:41337 every 8 connection(s)
I20260926 16:43:15.557255 10808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:43:15.564117 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 10808
I20260926 16:43:15.564401 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance
I20260926 16:43:15.565248 10171 external_mini_cluster.cc:1428] Setting key f75d0a3f19b6e71ac2fd940d79798d3a
Sep 26 16:43:15 dist-test-slave-wt2j krb5kdc[10702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440995, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.238.194@KRBTEST.COM; defaulting to no policy
I20260926 16:43:15.588277 10938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:15.560815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:50959 (local address 127.9.238.254:39693)
0926 16:43:15.561363 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:15.561369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:15.562863 (+ 1494us) server_negotiation.cc:407] Connection header received
0926 16:43:15.566186 (+ 3323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:15.566193 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:15.566312 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:15.566476 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:15.569112 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:15.570560 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:15.573105 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:15.573938 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:15.574157 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:15.579589 (+ 5432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:15.579600 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:15.579609 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:15.579675 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:15.583707 (+ 4032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:15.584982 (+ 1275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:15.584991 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:15.584997 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:15.585183 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:15.585801 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:15.585808 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:15.585814 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:15.586255 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:15.586439 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:15.587489 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0926 16:43:15.587848 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":255,"threads_started":1}
I20260926 16:43:15.589572 10939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:15.561473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:39693 (local address 127.9.238.193:50959)
0926 16:43:15.562638 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:15.562707 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:15.565701 (+ 2994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:15.567005 (+ 1304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:15.567022 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:15.568010 (+ 988us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:15.568825 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:15.568846 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:15.570793 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:15.570798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:15.572848 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:15.572870 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:15.574278 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:15.574287 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:15.574537 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:15.575334 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:15.575359 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:15.579072 (+ 3713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:15.583993 (+ 4921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:15.584006 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:15.584030 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:15.584750 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:15.585343 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:15.585347 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:15.585349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:15.585606 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:15.586411 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:15.586419 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:15.586911 (+ 492us) client_negotiation.cc:769] Sending connection context
0926 16:43:15.587280 (+ 369us) client_negotiation.cc:240] Negotiation successful
0926 16:43:15.588062 (+ 782us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":312,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.194@KRBTEST.COM" created.
I20260926 16:43:15.591786 10936 heartbeater.cc:347] Connected to a master server at 127.9.238.254:39693
I20260926 16:43:15.592238 10936 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:15.593149 10936 heartbeater.cc:511] Master 127.9.238.254:39693 requested a full tablet report, sending...
I20260926 16:43:15.595852 10747 ts_manager.cc:194] Registered new tserver with Master: d6f6fb3f5bd84a3fb1060f27086dd946 (127.9.238.193:41337)
I20260926 16:43:15.597689 10747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.193@KRBTEST.COM'} at 127.9.238.193:50959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
I20260926 16:43:15.620997 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:0
--local_ip_for_outbound_sockets=127.9.238.194
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:39693
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:46567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:15.978317 10942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:15.978631 10942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:15.978787 10942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:15.986320 10942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:15.986456 10942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:15.986653 10942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:43:15.997864 10942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:46567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:39693
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:15.999845 10942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:16.002285 10942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:16.019135 10947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:16.019043 10950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:16.019048 10948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:16.020040 10942 server_base.cc:1034] running on GCE node
I20260926 16:43:16.020741 10942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:16.021926 10942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:16.023218 10942 hybrid_clock.cc:648] HybridClock initialized: now 1790440996023153 us; error 58 us; skew 500 ppm
I20260926 16:43:16.023677 10942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:16 dist-test-slave-wt2j krb5kdc[10702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440996, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:16.027482 10942 init.cc:377] Logged in from keytab as kudu/127.9.238.194@KRBTEST.COM (short username kudu)
I20260926 16:43:16.029315 10942 webserver.cc:460] Webserver started at http://127.9.238.194:40539/ using document root <none> and password file <none>
I20260926 16:43:16.030107 10942 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:16.030270 10942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:16.030629 10942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:16.033840 10942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-1/data/instance:
uuid: "4501de12935e42c092b0cdac15fe5e05"
format_stamp: "Formatted at 2026-09-26 16:43:16 on dist-test-slave-wt2j"
server_key: "2ea087565e20b370b170070e3d3dde8b"
server_key_iv: "fe0638db39bafc4c36651936fd375b39"
server_key_version: "encryptionkey@0"
I20260926 16:43:16.034744 10942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance:
uuid: "4501de12935e42c092b0cdac15fe5e05"
format_stamp: "Formatted at 2026-09-26 16:43:16 on dist-test-slave-wt2j"
server_key: "2ea087565e20b370b170070e3d3dde8b"
server_key_iv: "fe0638db39bafc4c36651936fd375b39"
server_key_version: "encryptionkey@0"
I20260926 16:43:16.040773 10942 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260926 16:43:16.045286 10957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:16.047187 10942 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 16:43:16.047430 10942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "4501de12935e42c092b0cdac15fe5e05"
format_stamp: "Formatted at 2026-09-26 16:43:16 on dist-test-slave-wt2j"
server_key: "2ea087565e20b370b170070e3d3dde8b"
server_key_iv: "fe0638db39bafc4c36651936fd375b39"
server_key_version: "encryptionkey@0"
I20260926 16:43:16.047760 10942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:16.062557 10942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:16.067657 10942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:16.067997 10942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:16.069217 10942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:16.070936 10942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:16.071022 10942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:16.071193 10942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:16.071270 10942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:16.115314 10942 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:40329
I20260926 16:43:16.115435 11069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:40329 every 8 connection(s)
I20260926 16:43:16.116760 10942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:43:16.123435 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 10942
I20260926 16:43:16.123673 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance
I20260926 16:43:16.124171 10171 external_mini_cluster.cc:1428] Setting key 048aad7c740a995a9b5a2d241717f4a1
Sep 26 16:43:16 dist-test-slave-wt2j krb5kdc[10702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440996, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.238.195@KRBTEST.COM; defaulting to no policy
I20260926 16:43:16.142531 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:16.120031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:51167 (local address 127.9.238.254:39693)
0926 16:43:16.120629 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:16.120637 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:16.121855 (+ 1218us) server_negotiation.cc:407] Connection header received
0926 16:43:16.124444 (+ 2589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:16.124452 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:16.124574 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:16.124736 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:16.126975 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:16.127902 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:16.129746 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:16.130243 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:16.130407 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:16.134867 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:16.134874 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:16.134878 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:16.134922 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:16.138220 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:16.139264 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:16.139271 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:16.139283 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:16.139453 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:16.140080 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:16.140088 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:16.140091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:16.140464 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:16.140643 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:16.141716 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0926 16:43:16.142012 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":267,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.195@KRBTEST.COM" created.
I20260926 16:43:16.144059 11073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:16.120589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:39693 (local address 127.9.238.194:51167)
0926 16:43:16.121625 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:16.121679 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:16.123992 (+ 2313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:16.125070 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:16.125086 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:16.125865 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:16.126761 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:16.126784 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:16.128073 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:16.128087 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:16.129562 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:16.129577 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:16.130530 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:16.130538 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:16.130841 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:16.131831 (+ 990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:16.131860 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:16.134516 (+ 2656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:16.138450 (+ 3934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:16.138470 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:16.138494 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:16.139088 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:16.139640 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:16.139646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:16.139651 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:16.139960 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:16.140626 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:16.140637 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:16.141226 (+ 589us) client_negotiation.cc:769] Sending connection context
0926 16:43:16.141590 (+ 364us) client_negotiation.cc:240] Negotiation successful
0926 16:43:16.142342 (+ 752us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":347,"threads_started":1}
I20260926 16:43:16.146553 11070 heartbeater.cc:347] Connected to a master server at 127.9.238.254:39693
I20260926 16:43:16.147085 11070 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:16.148185 11070 heartbeater.cc:511] Master 127.9.238.254:39693 requested a full tablet report, sending...
I20260926 16:43:16.150365 10747 ts_manager.cc:194] Registered new tserver with Master: 4501de12935e42c092b0cdac15fe5e05 (127.9.238.194:40329)
I20260926 16:43:16.151362 10747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.194@KRBTEST.COM'} at 127.9.238.194:51167
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
I20260926 16:43:16.167935 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:0
--local_ip_for_outbound_sockets=127.9.238.195
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:39693
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:46567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:16.501698 11076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:16.501996 11076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:16.502147 11076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:16.509685 11076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:16.509811 11076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:16.509987 11076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:43:16.520221 11076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:46567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:39693
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:16.521661 11076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:16.523015 11076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:16.538965 11081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:16.538918 11082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:16.539155 11084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:16.540393 11076 server_base.cc:1034] running on GCE node
I20260926 16:43:16.541168 11076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:16.542341 11076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:16.543682 11076 hybrid_clock.cc:648] HybridClock initialized: now 1790440996543612 us; error 59 us; skew 500 ppm
I20260926 16:43:16.544070 11076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:16 dist-test-slave-wt2j krb5kdc[10702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440996, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:16.547919 11076 init.cc:377] Logged in from keytab as kudu/127.9.238.195@KRBTEST.COM (short username kudu)
I20260926 16:43:16.549816 11076 webserver.cc:460] Webserver started at http://127.9.238.195:37295/ using document root <none> and password file <none>
I20260926 16:43:16.550645 11076 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:16.550812 11076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:16.551178 11076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:16.554617 11076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-2/data/instance:
uuid: "43fab70c88d24b098d6f1f84a2d9b44a"
format_stamp: "Formatted at 2026-09-26 16:43:16 on dist-test-slave-wt2j"
server_key: "2a4df07922b8fccfa8f6ef4cd4d6ad13"
server_key_iv: "2f289b1d56049e13f4b0bc1a9b8ee60a"
server_key_version: "encryptionkey@0"
I20260926 16:43:16.555682 11076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance:
uuid: "43fab70c88d24b098d6f1f84a2d9b44a"
format_stamp: "Formatted at 2026-09-26 16:43:16 on dist-test-slave-wt2j"
server_key: "2a4df07922b8fccfa8f6ef4cd4d6ad13"
server_key_iv: "2f289b1d56049e13f4b0bc1a9b8ee60a"
server_key_version: "encryptionkey@0"
I20260926 16:43:16.562577 11076 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 16:43:16.567026 11091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:16.568774 11076 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:43:16.568949 11076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "43fab70c88d24b098d6f1f84a2d9b44a"
format_stamp: "Formatted at 2026-09-26 16:43:16 on dist-test-slave-wt2j"
server_key: "2a4df07922b8fccfa8f6ef4cd4d6ad13"
server_key_iv: "2f289b1d56049e13f4b0bc1a9b8ee60a"
server_key_version: "encryptionkey@0"
I20260926 16:43:16.569159 11076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:16.601207 10936 heartbeater.cc:503] Master 127.9.238.254:39693 was elected leader, sending a full tablet report...
I20260926 16:43:16.602466 11076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:16.607856 11076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:16.608210 11076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:16.609853 11076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:16.611474 11076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:16.611641 11076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 16:43:16.611761 11076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:16.611799 11076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:16.656306 11076 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:35101
I20260926 16:43:16.656401 11203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:35101 every 8 connection(s)
I20260926 16:43:16.657838 11076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:43:16.659842 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 11076
I20260926 16:43:16.660064 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance
I20260926 16:43:16.660490 10171 external_mini_cluster.cc:1428] Setting key 0067da530892d6e582dcc566fefc8739
Sep 26 16:43:16 dist-test-slave-wt2j krb5kdc[10702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440996, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:16.688822 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:16.663674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:50295 (local address 127.9.238.254:39693)
0926 16:43:16.664282 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:16.664289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:16.665932 (+ 1643us) server_negotiation.cc:407] Connection header received
0926 16:43:16.669243 (+ 3311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:16.669252 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:16.669363 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:16.669504 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:16.673175 (+ 3671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:16.674259 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:16.676704 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:16.677177 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:16.677345 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:16.681917 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:16.681923 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:16.681928 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:16.681970 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:16.685022 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:16.685921 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:16.685925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:16.685928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:16.686071 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:16.686621 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:16.686625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:16.686627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:16.686952 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:16.687095 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:16.688049 (+ 954us) server_negotiation.cc:299] Negotiation successful
0926 16:43:16.688372 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":308,"threads_started":1}
I20260926 16:43:16.689826 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:16.664333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:39693 (local address 127.9.238.195:50295)
0926 16:43:16.665510 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:16.665712 (+ 202us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:16.668719 (+ 3007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:16.670672 (+ 1953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:16.670687 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:16.671765 (+ 1078us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:16.672710 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:16.672740 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:16.674609 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:16.674614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:16.676483 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:16.676504 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:16.677482 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:16.677491 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:16.677799 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:16.678585 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:16.678612 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:16.681588 (+ 2976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:16.685211 (+ 3623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:16.685221 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:16.685236 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:16.685766 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:16.686237 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:16.686242 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:16.686245 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:16.686478 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:16.687117 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:16.687124 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:16.687605 (+ 481us) client_negotiation.cc:769] Sending connection context
0926 16:43:16.687888 (+ 283us) client_negotiation.cc:240] Negotiation successful
0926 16:43:16.688416 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":450,"threads_started":1}
I20260926 16:43:16.691454 11204 heartbeater.cc:347] Connected to a master server at 127.9.238.254:39693
I20260926 16:43:16.691951 11204 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:16.692972 11204 heartbeater.cc:511] Master 127.9.238.254:39693 requested a full tablet report, sending...
I20260926 16:43:16.694839 10746 ts_manager.cc:194] Registered new tserver with Master: 43fab70c88d24b098d6f1f84a2d9b44a (127.9.238.195:35101)
I20260926 16:43:16.695996 10746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.195@KRBTEST.COM'} at 127.9.238.195:50295
I20260926 16:43:16.704862 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:43:16.726874 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:16.712123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39650 (local address 127.9.238.254:39693)
0926 16:43:16.712367 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:16.712373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:16.712676 (+ 303us) server_negotiation.cc:407] Connection header received
0926 16:43:16.712922 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:16.712927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:16.713018 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:16.713151 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:16.714795 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:16.715888 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:16.717342 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:16.717852 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:16.717982 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:16.720551 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:16.720556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:16.720561 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:16.720601 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:16.723747 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:16.724602 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:16.724606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:16.724609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:16.724770 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:16.725185 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:16.725189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:16.725191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:16.725688 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:16.725935 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:16.726307 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 16:43:16.726592 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 16:43:16.734746 10746 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:43:16.742149 10746 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:43:16.782567 11217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:16.757833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:35101 (local address 127.0.0.1:55824)
0926 16:43:16.770399 (+ 12566us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:16.770476 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:16.770780 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:16.772817 (+ 2037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:16.772830 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:16.772918 (+ 88us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:16.773792 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:16.773812 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:16.777294 (+ 3482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:16.777302 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:16.779856 (+ 2554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:16.779872 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:16.781408 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:16.781415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:16.781605 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:16.781668 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 16:43:16.781812 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 16:43:16.782059 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12127,"mutex_wait_us":7,"spinlock_wait_cycles":191360,"thread_start_us":11943,"threads_started":1}
I20260926 16:43:16.782598 11219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:16.758213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55824 (local address 127.9.238.195:35101)
0926 16:43:16.771491 (+ 13278us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:16.771501 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:16.771565 (+ 64us) server_negotiation.cc:407] Connection header received
0926 16:43:16.771701 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:16.771711 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:16.772144 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:16.775701 (+ 3557us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:16.776034 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:16.777083 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:16.780085 (+ 3002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:16.781143 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:16.781314 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:16.781610 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:16.781953 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 16:43:16.782224 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13038,"thread_start_us":14172,"threads_started":1}
I20260926 16:43:16.784521 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:16.771648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:41337 (local address 127.0.0.1:60708)
0926 16:43:16.775829 (+ 4181us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:16.775843 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:16.775993 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:16.776573 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:16.776578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:16.776602 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:16.777001 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:16.777013 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:16.779031 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:16.779037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:16.781215 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:16.781228 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:16.783755 (+ 2527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:16.783760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:16.783876 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:16.783902 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 16:43:16.784014 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 16:43:16.784154 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4071,"thread_start_us":3930,"threads_started":1}
I20260926 16:43:16.784680 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:16.771774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60708 (local address 127.9.238.193:41337)
0926 16:43:16.775534 (+ 3760us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:16.775568 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:16.775950 (+ 382us) server_negotiation.cc:407] Connection header received
0926 16:43:16.776173 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:16.776181 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:16.776382 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:16.776586 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:16.777218 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:16.778858 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:16.781437 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:16.783052 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:16.783202 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:16.783427 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:16.784169 (+ 742us) server_negotiation.cc:299] Negotiation successful
0926 16:43:16.784339 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3568,"thread_start_us":3378,"threads_started":1}
I20260926 16:43:16.785692 11218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:16.757830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:40329 (local address 127.0.0.1:52498)
0926 16:43:16.771448 (+ 13618us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:16.771467 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:16.771632 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:16.772920 (+ 1288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:16.772929 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:16.773061 (+ 132us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:16.774396 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:16.774413 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:16.778409 (+ 3996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:16.778416 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:16.781994 (+ 3578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:16.782007 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:16.784700 (+ 2693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:16.784706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:16.784889 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:16.784931 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 16:43:16.785033 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 16:43:16.785195 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13510,"mutex_wait_us":962,"thread_start_us":13182,"threads_started":1}
I20260926 16:43:16.786803 11220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:16.758260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52498 (local address 127.9.238.194:40329)
0926 16:43:16.769723 (+ 11463us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:16.769736 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:16.771767 (+ 2031us) server_negotiation.cc:407] Connection header received
0926 16:43:16.771903 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:16.771914 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:16.772324 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:16.772587 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:16.774701 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:16.778265 (+ 3564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:16.782246 (+ 3981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:16.784179 (+ 1933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:16.784339 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:16.784558 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:16.786236 (+ 1678us) server_negotiation.cc:299] Negotiation successful
0926 16:43:16.786453 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11266,"thread_start_us":11102,"threads_started":1}
I20260926 16:43:16.788383 11139 tablet_service.cc:1467] Processing CreateTablet for tablet 228fe74bb5f14c10a4766bddd1cb8427 (DEFAULT_TABLE table=test-table [id=1d2e4da107a342f5836c1bbb8de1e81f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:16.790027 11139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 228fe74bb5f14c10a4766bddd1cb8427. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:16.790360 11005 tablet_service.cc:1467] Processing CreateTablet for tablet 228fe74bb5f14c10a4766bddd1cb8427 (DEFAULT_TABLE table=test-table [id=1d2e4da107a342f5836c1bbb8de1e81f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:16.790359 10871 tablet_service.cc:1467] Processing CreateTablet for tablet 228fe74bb5f14c10a4766bddd1cb8427 (DEFAULT_TABLE table=test-table [id=1d2e4da107a342f5836c1bbb8de1e81f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:16.792084 11005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 228fe74bb5f14c10a4766bddd1cb8427. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:16.792089 10871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 228fe74bb5f14c10a4766bddd1cb8427. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:16.803779 11223 tablet_bootstrap.cc:492] T 228fe74bb5f14c10a4766bddd1cb8427 P 43fab70c88d24b098d6f1f84a2d9b44a: Bootstrap starting.
I20260926 16:43:16.808868 11224 tablet_bootstrap.cc:492] T 228fe74bb5f14c10a4766bddd1cb8427 P 4501de12935e42c092b0cdac15fe5e05: Bootstrap starting.
I20260926 16:43:16.810251 11223 tablet_bootstrap.cc:654] T 228fe74bb5f14c10a4766bddd1cb8427 P 43fab70c88d24b098d6f1f84a2d9b44a: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:16.810387 11225 tablet_bootstrap.cc:492] T 228fe74bb5f14c10a4766bddd1cb8427 P d6f6fb3f5bd84a3fb1060f27086dd946: Bootstrap starting.
I20260926 16:43:16.812106 11223 log.cc:824] T 228fe74bb5f14c10a4766bddd1cb8427 P 43fab70c88d24b098d6f1f84a2d9b44a: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:16.815125 11224 tablet_bootstrap.cc:654] T 228fe74bb5f14c10a4766bddd1cb8427 P 4501de12935e42c092b0cdac15fe5e05: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:16.815366 11225 tablet_bootstrap.cc:654] T 228fe74bb5f14c10a4766bddd1cb8427 P d6f6fb3f5bd84a3fb1060f27086dd946: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:16.816926 11224 log.cc:824] T 228fe74bb5f14c10a4766bddd1cb8427 P 4501de12935e42c092b0cdac15fe5e05: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:16.817298 11225 log.cc:824] T 228fe74bb5f14c10a4766bddd1cb8427 P d6f6fb3f5bd84a3fb1060f27086dd946: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:16.828258 11223 tablet_bootstrap.cc:492] T 228fe74bb5f14c10a4766bddd1cb8427 P 43fab70c88d24b098d6f1f84a2d9b44a: No bootstrap required, opened a new log
I20260926 16:43:16.828790 11223 ts_tablet_manager.cc:1397] T 228fe74bb5f14c10a4766bddd1cb8427 P 43fab70c88d24b098d6f1f84a2d9b44a: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.006s
I20260926 16:43:16.830878 11224 tablet_bootstrap.cc:492] T 228fe74bb5f14c10a4766bddd1cb8427 P 4501de12935e42c092b0cdac15fe5e05: No bootstrap required, opened a new log
I20260926 16:43:16.831300 11224 ts_tablet_manager.cc:1397] T 228fe74bb5f14c10a4766bddd1cb8427 P 4501de12935e42c092b0cdac15fe5e05: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.013s
I20260926 16:43:16.840546 11225 tablet_bootstrap.cc:492] T 228fe74bb5f14c10a4766bddd1cb8427 P d6f6fb3f5bd84a3fb1060f27086dd946: No bootstrap required, opened a new log
I20260926 16:43:16.840958 11225 ts_tablet_manager.cc:1397] T 228fe74bb5f14c10a4766bddd1cb8427 P d6f6fb3f5bd84a3fb1060f27086dd946: Time spent bootstrapping tablet: real 0.031s user 0.015s sys 0.013s
I20260926 16:43:16.841571 11224 raft_consensus.cc:348] T 228fe74bb5f14c10a4766bddd1cb8427 P 4501de12935e42c092b0cdac15fe5e05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43fab70c88d24b098d6f1f84a2d9b44a" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35101 } } peers { permanent_uuid: "4501de12935e42c092b0cdac15fe5e05" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 40329 } } peers { permanent_uuid: "d6f6fb3f5bd84a3fb1060f27086dd946" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41337 } }
I20260926 16:43:16.842012 11224 raft_consensus.cc:374] T 228fe74bb5f14c10a4766bddd1cb8427 P 4501de12935e42c092b0cdac15fe5e05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:16.842104 11224 raft_consensus.cc:729] T 228fe74bb5f14c10a4766bddd1cb8427 P 4501de12935e42c092b0cdac15fe5e05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4501de12935e42c092b0cdac15fe5e05, State: Initialized, Role: FOLLOWER
I20260926 16:43:16.842782 11224 consensus_queue.cc:260] T 228fe74bb5f14c10a4766bddd1cb8427 P 4501de12935e42c092b0cdac15fe5e05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43fab70c88d24b098d6f1f84a2d9b44a" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35101 } } peers { permanent_uuid: "4501de12935e42c092b0cdac15fe5e05" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 40329 } } peers { permanent_uuid: "d6f6fb3f5bd84a3fb1060f27086dd946" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41337 } }
I20260926 16:43:16.844110 11070 heartbeater.cc:503] Master 127.9.238.254:39693 was elected leader, sending a full tablet report...
I20260926 16:43:16.844660 11224 ts_tablet_manager.cc:1428] T 228fe74bb5f14c10a4766bddd1cb8427 P 4501de12935e42c092b0cdac15fe5e05: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260926 16:43:16.843371 11223 raft_consensus.cc:348] T 228fe74bb5f14c10a4766bddd1cb8427 P 43fab70c88d24b098d6f1f84a2d9b44a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43fab70c88d24b098d6f1f84a2d9b44a" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35101 } } peers { permanent_uuid: "4501de12935e42c092b0cdac15fe5e05" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 40329 } } peers { permanent_uuid: "d6f6fb3f5bd84a3fb1060f27086dd946" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41337 } }
I20260926 16:43:16.846311 11223 raft_consensus.cc:374] T 228fe74bb5f14c10a4766bddd1cb8427 P 43fab70c88d24b098d6f1f84a2d9b44a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:16.846436 11223 raft_consensus.cc:729] T 228fe74bb5f14c10a4766bddd1cb8427 P 43fab70c88d24b098d6f1f84a2d9b44a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43fab70c88d24b098d6f1f84a2d9b44a, State: Initialized, Role: FOLLOWER
I20260926 16:43:16.847280 11223 consensus_queue.cc:260] T 228fe74bb5f14c10a4766bddd1cb8427 P 43fab70c88d24b098d6f1f84a2d9b44a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43fab70c88d24b098d6f1f84a2d9b44a" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35101 } } peers { permanent_uuid: "4501de12935e42c092b0cdac15fe5e05" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 40329 } } peers { permanent_uuid: "d6f6fb3f5bd84a3fb1060f27086dd946" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41337 } }
I20260926 16:43:16.848727 11204 heartbeater.cc:503] Master 127.9.238.254:39693 was elected leader, sending a full tablet report...
I20260926 16:43:16.852592 11223 ts_tablet_manager.cc:1428] T 228fe74bb5f14c10a4766bddd1cb8427 P 43fab70c88d24b098d6f1f84a2d9b44a: Time spent starting tablet: real 0.024s user 0.011s sys 0.008s
I20260926 16:43:16.852461 11225 raft_consensus.cc:348] T 228fe74bb5f14c10a4766bddd1cb8427 P d6f6fb3f5bd84a3fb1060f27086dd946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43fab70c88d24b098d6f1f84a2d9b44a" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35101 } } peers { permanent_uuid: "4501de12935e42c092b0cdac15fe5e05" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 40329 } } peers { permanent_uuid: "d6f6fb3f5bd84a3fb1060f27086dd946" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41337 } }
I20260926 16:43:16.853044 11225 raft_consensus.cc:374] T 228fe74bb5f14c10a4766bddd1cb8427 P d6f6fb3f5bd84a3fb1060f27086dd946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:16.853160 11225 raft_consensus.cc:729] T 228fe74bb5f14c10a4766bddd1cb8427 P d6f6fb3f5bd84a3fb1060f27086dd946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6f6fb3f5bd84a3fb1060f27086dd946, State: Initialized, Role: FOLLOWER
I20260926 16:43:16.854022 11225 consensus_queue.cc:260] T 228fe74bb5f14c10a4766bddd1cb8427 P d6f6fb3f5bd84a3fb1060f27086dd946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43fab70c88d24b098d6f1f84a2d9b44a" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35101 } } peers { permanent_uuid: "4501de12935e42c092b0cdac15fe5e05" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 40329 } } peers { permanent_uuid: "d6f6fb3f5bd84a3fb1060f27086dd946" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41337 } }
I20260926 16:43:16.856287 11225 ts_tablet_manager.cc:1428] T 228fe74bb5f14c10a4766bddd1cb8427 P d6f6fb3f5bd84a3fb1060f27086dd946: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
W20260926 16:43:16.869144 11071 tablet.cc:2369] T 228fe74bb5f14c10a4766bddd1cb8427 P 4501de12935e42c092b0cdac15fe5e05: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:43:16.912228 11205 tablet.cc:2369] T 228fe74bb5f14c10a4766bddd1cb8427 P 43fab70c88d24b098d6f1f84a2d9b44a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:43:17.060446 10937 tablet.cc:2369] T 228fe74bb5f14c10a4766bddd1cb8427 P d6f6fb3f5bd84a3fb1060f27086dd946: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:43:17.159183 11229 raft_consensus.cc:482] T 228fe74bb5f14c10a4766bddd1cb8427 P 4501de12935e42c092b0cdac15fe5e05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:43:17.159713 11229 raft_consensus.cc:504] T 228fe74bb5f14c10a4766bddd1cb8427 P 4501de12935e42c092b0cdac15fe5e05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43fab70c88d24b098d6f1f84a2d9b44a" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35101 } } peers { permanent_uuid: "4501de12935e42c092b0cdac15fe5e05" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 40329 } } peers { permanent_uuid: "d6f6fb3f5bd84a3fb1060f27086dd946" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41337 } }
I20260926 16:43:17.161948 11229 leader_election.cc:288] T 228fe74bb5f14c10a4766bddd1cb8427 P 4501de12935e42c092b0cdac15fe5e05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43fab70c88d24b098d6f1f84a2d9b44a (127.9.238.195:35101), d6f6fb3f5bd84a3fb1060f27086dd946 (127.9.238.193:41337)
I20260926 16:43:17.170405 11219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:17.162581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:57607 (local address 127.9.238.195:35101)
0926 16:43:17.162894 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:17.162905 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:17.163402 (+ 497us) server_negotiation.cc:407] Connection header received
0926 16:43:17.163794 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:17.163802 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:17.163903 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:17.163978 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:17.164849 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:17.165937 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:17.168330 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:17.169367 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:17.169501 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:17.169566 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:17.170089 (+ 523us) server_negotiation.cc:299] Negotiation successful
0926 16:43:17.170204 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 16:43:17.170761 11232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:17.162446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:35101 (local address 127.9.238.194:57607)
0926 16:43:17.163217 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:17.163241 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:17.163556 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:17.164212 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:17.164215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:17.164241 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:17.164660 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:17.164675 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:17.166089 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:17.166096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:17.168168 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:17.168181 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:17.169644 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:17.169652 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:17.169857 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:17.169901 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 16:43:17.170110 (+ 209us) client_negotiation.cc:240] Negotiation successful
0926 16:43:17.170324 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":449,"threads_started":1}
I20260926 16:43:17.171967 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:17.163761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:47475 (local address 127.9.238.193:41337)
0926 16:43:17.164006 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:17.164012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:17.164377 (+ 365us) server_negotiation.cc:407] Connection header received
0926 16:43:17.164588 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:17.164594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:17.164681 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:17.164782 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:17.165586 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:17.166743 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:17.169076 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:17.170806 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:17.170999 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:17.171101 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:17.171507 (+ 406us) server_negotiation.cc:299] Negotiation successful
0926 16:43:17.171689 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 16:43:17.171998 11233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:17.163388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:41337 (local address 127.9.238.194:47475)
0926 16:43:17.164242 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:17.164260 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:17.164436 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:17.164832 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:17.164837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:17.164863 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:17.165358 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:17.165374 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:17.166904 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:17.166908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:17.168828 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:17.168842 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:17.171182 (+ 2340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:17.171187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:17.171330 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:17.171355 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 16:43:17.171477 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 16:43:17.171651 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":504,"threads_started":1}
I20260926 16:43:17.171988 11159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228fe74bb5f14c10a4766bddd1cb8427" candidate_uuid: "4501de12935e42c092b0cdac15fe5e05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43fab70c88d24b098d6f1f84a2d9b44a" is_pre_election: true
I20260926 16:43:17.172554 11159 raft_consensus.cc:2454] T 228fe74bb5f14c10a4766bddd1cb8427 P 43fab70c88d24b098d6f1f84a2d9b44a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4501de12935e42c092b0cdac15fe5e05 in term 0.
I20260926 16:43:17.173228 10958 leader_election.cc:302] T 228fe74bb5f14c10a4766bddd1cb8427 P 4501de12935e42c092b0cdac15fe5e05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43fab70c88d24b098d6f1f84a2d9b44a, 4501de12935e42c092b0cdac15fe5e05; no voters:
I20260926 16:43:17.173209 10891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228fe74bb5f14c10a4766bddd1cb8427" candidate_uuid: "4501de12935e42c092b0cdac15fe5e05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6f6fb3f5bd84a3fb1060f27086dd946" is_pre_election: true
I20260926 16:43:17.173691 11229 raft_consensus.cc:2781] T 228fe74bb5f14c10a4766bddd1cb8427 P 4501de12935e42c092b0cdac15fe5e05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:43:17.173836 11229 raft_consensus.cc:482] T 228fe74bb5f14c10a4766bddd1cb8427 P 4501de12935e42c092b0cdac15fe5e05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:43:17.173861 10891 raft_consensus.cc:2454] T 228fe74bb5f14c10a4766bddd1cb8427 P d6f6fb3f5bd84a3fb1060f27086dd946 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4501de12935e42c092b0cdac15fe5e05 in term 0.
I20260926 16:43:17.173982 11229 raft_consensus.cc:3037] T 228fe74bb5f14c10a4766bddd1cb8427 P 4501de12935e42c092b0cdac15fe5e05 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:17.176671 11229 raft_consensus.cc:504] T 228fe74bb5f14c10a4766bddd1cb8427 P 4501de12935e42c092b0cdac15fe5e05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43fab70c88d24b098d6f1f84a2d9b44a" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35101 } } peers { permanent_uuid: "4501de12935e42c092b0cdac15fe5e05" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 40329 } } peers { permanent_uuid: "d6f6fb3f5bd84a3fb1060f27086dd946" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41337 } }
I20260926 16:43:17.177353 11229 leader_election.cc:288] T 228fe74bb5f14c10a4766bddd1cb8427 P 4501de12935e42c092b0cdac15fe5e05 [CANDIDATE]: Term 1 election: Requested vote from peers 43fab70c88d24b098d6f1f84a2d9b44a (127.9.238.195:35101), d6f6fb3f5bd84a3fb1060f27086dd946 (127.9.238.193:41337)
I20260926 16:43:17.177978 11159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228fe74bb5f14c10a4766bddd1cb8427" candidate_uuid: "4501de12935e42c092b0cdac15fe5e05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43fab70c88d24b098d6f1f84a2d9b44a"
I20260926 16:43:17.178059 10891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228fe74bb5f14c10a4766bddd1cb8427" candidate_uuid: "4501de12935e42c092b0cdac15fe5e05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6f6fb3f5bd84a3fb1060f27086dd946"
I20260926 16:43:17.178182 11159 raft_consensus.cc:3037] T 228fe74bb5f14c10a4766bddd1cb8427 P 43fab70c88d24b098d6f1f84a2d9b44a [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:17.178450 10891 raft_consensus.cc:3037] T 228fe74bb5f14c10a4766bddd1cb8427 P d6f6fb3f5bd84a3fb1060f27086dd946 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:17.180668 11159 raft_consensus.cc:2454] T 228fe74bb5f14c10a4766bddd1cb8427 P 43fab70c88d24b098d6f1f84a2d9b44a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4501de12935e42c092b0cdac15fe5e05 in term 1.
I20260926 16:43:17.181335 10958 leader_election.cc:302] T 228fe74bb5f14c10a4766bddd1cb8427 P 4501de12935e42c092b0cdac15fe5e05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43fab70c88d24b098d6f1f84a2d9b44a, 4501de12935e42c092b0cdac15fe5e05; no voters:
I20260926 16:43:17.181561 10891 raft_consensus.cc:2454] T 228fe74bb5f14c10a4766bddd1cb8427 P d6f6fb3f5bd84a3fb1060f27086dd946 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4501de12935e42c092b0cdac15fe5e05 in term 1.
I20260926 16:43:17.181751 11229 raft_consensus.cc:2781] T 228fe74bb5f14c10a4766bddd1cb8427 P 4501de12935e42c092b0cdac15fe5e05 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:17.182387 11229 raft_consensus.cc:686] T 228fe74bb5f14c10a4766bddd1cb8427 P 4501de12935e42c092b0cdac15fe5e05 [term 1 LEADER]: Becoming Leader. State: Replica: 4501de12935e42c092b0cdac15fe5e05, State: Running, Role: LEADER
I20260926 16:43:17.183024 11229 consensus_queue.cc:237] T 228fe74bb5f14c10a4766bddd1cb8427 P 4501de12935e42c092b0cdac15fe5e05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43fab70c88d24b098d6f1f84a2d9b44a" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35101 } } peers { permanent_uuid: "4501de12935e42c092b0cdac15fe5e05" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 40329 } } peers { permanent_uuid: "d6f6fb3f5bd84a3fb1060f27086dd946" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41337 } }
I20260926 16:43:17.188617 10746 catalog_manager.cc:5495] T 228fe74bb5f14c10a4766bddd1cb8427 P 4501de12935e42c092b0cdac15fe5e05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4501de12935e42c092b0cdac15fe5e05 (127.9.238.194). New cstate: current_term: 1 leader_uuid: "4501de12935e42c092b0cdac15fe5e05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43fab70c88d24b098d6f1f84a2d9b44a" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4501de12935e42c092b0cdac15fe5e05" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 40329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6f6fb3f5bd84a3fb1060f27086dd946" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41337 } health_report { overall_health: UNKNOWN } } }
Sep 26 16:43:17 dist-test-slave-wt2j krb5kdc[10702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.193@KRBTEST.COM
I20260926 16:43:17.258589 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:17.239758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60720 (local address 127.9.238.193:41337)
0926 16:43:17.240106 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:17.240117 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:17.240578 (+ 461us) server_negotiation.cc:407] Connection header received
0926 16:43:17.240856 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:17.240868 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:17.240949 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:17.241082 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:17.242799 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:17.243841 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:17.245362 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:17.245931 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:17.246131 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:17.251401 (+ 5270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:17.251424 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:17.251447 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:17.251505 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:17.255152 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:17.256068 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:17.256076 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:17.256079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:17.256247 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:17.256675 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:17.256684 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:17.256687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:17.257214 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:17.257558 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:17.257898 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 16:43:17.258162 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
Sep 26 16:43:17 dist-test-slave-wt2j krb5kdc[10702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440997, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 16:43:17 dist-test-slave-wt2j krb5kdc[10702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440997, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.238.193@KRBTEST.COM
I20260926 16:43:17.313225 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:17.298071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60734 (local address 127.9.238.193:41337)
0926 16:43:17.298403 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:17.298410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:17.298752 (+ 342us) server_negotiation.cc:407] Connection header received
0926 16:43:17.299010 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:17.299017 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:17.299098 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:17.299215 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:17.300735 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:17.301500 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:17.302765 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:17.303174 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:17.303315 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:17.307387 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:17.307392 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:17.307396 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:17.307430 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:17.310512 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:17.311197 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:17.311204 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:17.311212 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:17.311331 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:17.311770 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:17.311776 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:17.311779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:17.312055 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:17.312317 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:17.312642 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 16:43:17.312865 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
Sep 26 16:43:17 dist-test-slave-wt2j krb5kdc[10702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 16:43:17 dist-test-slave-wt2j krb5kdc[10702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.193@KRBTEST.COM
I20260926 16:43:17.365437 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:17.350029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60742 (local address 127.9.238.193:41337)
0926 16:43:17.350242 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:17.350251 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:17.350708 (+ 457us) server_negotiation.cc:407] Connection header received
0926 16:43:17.350945 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:17.350951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:17.351021 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:17.351130 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:17.352628 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:17.353389 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:17.354587 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:17.355003 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:17.355136 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:17.360014 (+ 4878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:17.360020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:17.360023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:17.360061 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:17.362708 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:17.363372 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:17.363377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:17.363381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:17.363552 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:17.363967 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:17.363973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:17.363976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:17.364265 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:17.364515 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:17.364820 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 16:43:17.365046 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 16:43:17.367213 10851 tablet_service.cc:2908] Rejecting scan request for tablet 228fe74bb5f14c10a4766bddd1cb8427: Uninitialized: clean time has not yet been initialized
I20260926 16:43:17.371620 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 10808
I20260926 16:43:17.385848 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:17.372629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60748 (local address 127.9.238.193:41337)
0926 16:43:17.372814 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:17.372819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:17.373236 (+ 417us) server_negotiation.cc:407] Connection header received
0926 16:43:17.373453 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:17.373457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:17.373538 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:17.373641 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:17.375007 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:17.375861 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:17.377255 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:17.377646 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:17.377760 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:17.380426 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:17.380430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:17.380433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:17.380464 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:17.383079 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:17.384096 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:17.384100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:17.384103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:17.384273 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:17.384631 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:17.384634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:17.384636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:17.384899 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:17.385106 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:17.385444 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 16:43:17.385605 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 16:43:17.386837 10931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:43:17.482690 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 10808
I20260926 16:43:17.501084 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 10942
Sep 26 16:43:17 dist-test-slave-wt2j krb5kdc[10702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.194@KRBTEST.COM
I20260926 16:43:17.518741 11248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:17.502235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52502 (local address 127.9.238.194:40329)
0926 16:43:17.502850 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:17.502855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:17.502876 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:43:17.502950 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:17.502954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:17.503026 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:17.503143 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:17.504646 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:17.505432 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:17.506829 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:17.507377 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:17.507702 (+ 325us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:17.512583 (+ 4881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:17.512602 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:17.512624 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:17.512673 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:17.515826 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:17.516591 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:17.516599 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:17.516602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:17.516748 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:17.517073 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:17.517076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:17.517078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:17.517521 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:17.517783 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:17.518048 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 16:43:17.518246 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":367,"threads_started":1}
I20260926 16:43:17.519622 11065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:43:17.616554 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 10942
I20260926 16:43:17.634086 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 11076
Sep 26 16:43:17 dist-test-slave-wt2j krb5kdc[10702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.195@KRBTEST.COM
I20260926 16:43:17.650694 11219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:17.635108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55834 (local address 127.9.238.195:35101)
0926 16:43:17.635351 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:17.635359 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:17.635481 (+ 122us) server_negotiation.cc:407] Connection header received
0926 16:43:17.635801 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:17.635806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:17.635875 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:17.635991 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:17.637219 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:17.637946 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:17.639275 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:17.639826 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:17.639971 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:17.644525 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:17.644542 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:17.644561 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:17.644604 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:17.647911 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:17.648639 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:17.648654 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:17.648657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:17.648787 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:17.649159 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:17.649166 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:17.649169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:17.649529 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:17.649871 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:17.650097 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 16:43:17.650307 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 16:43:17.651639 11199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:43:17.750444 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 11076
I20260926 16:43:17.768042 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 10715
I20260926 16:43:17.769441 10777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:43:17.898110 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 10715
2026-09-26T16:43:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4972 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:43:18 dist-test-slave-wt2j krb5kdc[11254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:43:18 dist-test-slave-wt2j krb5kdc[11254](info): set up 2 sockets
Sep 26 16:43:18 dist-test-slave-wt2j krb5kdc[11254](info): commencing operation
krb5kdc: starting...
W20260926 16:43:19.037053 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:43:19 dist-test-slave-wt2j krb5kdc[11254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:43:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:43:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:43:19.252096 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:33509
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:46317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:33509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:19.654145 11267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:19.654695 11267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:19.654910 11267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:19.663250 11267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:43:19.663493 11267 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 16:43:19.663602 11267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:19.663728 11267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:19.663832 11267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:43:19.663939 11267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:43:19.676194 11267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:46317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:33509
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:33509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:19.678107 11267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:19.680097 11267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:19.695006 11272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:19.694986 11275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:19.695350 11273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:19.695968 11267 server_base.cc:1034] running on GCE node
I20260926 16:43:19.696933 11267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:19.698683 11267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:19.700042 11267 hybrid_clock.cc:648] HybridClock initialized: now 1790440999699944 us; error 47 us; skew 500 ppm
I20260926 16:43:19.700536 11267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:19 dist-test-slave-wt2j krb5kdc[11254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440999, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:19.709510 11267 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:43:19.711653 11267 webserver.cc:460] Webserver started at http://127.9.238.254:46359/ using document root <none> and password file <none>
I20260926 16:43:19.712482 11267 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:19.712591 11267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:19.712980 11267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:19.716451 11267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "17ee74c6a2a24bdcabd99741ead3b6fc"
format_stamp: "Formatted at 2026-09-26 16:43:19 on dist-test-slave-wt2j"
server_key: "0c7a1f5ae11aac939482a5e5c9aae99f"
server_key_iv: "4614d9dafe59823c5a3b631731bf4d24"
server_key_version: "encryptionkey@0"
I20260926 16:43:19.717559 11267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "17ee74c6a2a24bdcabd99741ead3b6fc"
format_stamp: "Formatted at 2026-09-26 16:43:19 on dist-test-slave-wt2j"
server_key: "0c7a1f5ae11aac939482a5e5c9aae99f"
server_key_iv: "4614d9dafe59823c5a3b631731bf4d24"
server_key_version: "encryptionkey@0"
I20260926 16:43:19.724431 11267 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 16:43:19.728935 11282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:19.730861 11267 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260926 16:43:19.731194 11267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "17ee74c6a2a24bdcabd99741ead3b6fc"
format_stamp: "Formatted at 2026-09-26 16:43:19 on dist-test-slave-wt2j"
server_key: "0c7a1f5ae11aac939482a5e5c9aae99f"
server_key_iv: "4614d9dafe59823c5a3b631731bf4d24"
server_key_version: "encryptionkey@0"
I20260926 16:43:19.731462 11267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:19.753180 11267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:19.758879 11267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:19.759244 11267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:19.786020 11267 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:33509
I20260926 16:43:19.786055 11343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:33509 every 8 connection(s)
I20260926 16:43:19.787678 11267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:43:19.792728 11344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:19.795137 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 11267
I20260926 16:43:19.795411 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:43:19.796240 10171 external_mini_cluster.cc:1428] Setting key 26503570cb3086b9bea88fcfe380c3b5
I20260926 16:43:19.809728 11344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17ee74c6a2a24bdcabd99741ead3b6fc: Bootstrap starting.
I20260926 16:43:19.815447 11344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17ee74c6a2a24bdcabd99741ead3b6fc: Neither blocks nor log segments found. Creating new log.
Sep 26 16:43:19 dist-test-slave-wt2j krb5kdc[11254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:19.816956 11344 log.cc:824] T 00000000000000000000000000000000 P 17ee74c6a2a24bdcabd99741ead3b6fc: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:19.821797 11344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17ee74c6a2a24bdcabd99741ead3b6fc: No bootstrap required, opened a new log
I20260926 16:43:19.828995 11347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:19.799399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59230 (local address 127.9.238.254:33509)
0926 16:43:19.800843 (+ 1444us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:19.800857 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:19.800911 (+ 54us) server_negotiation.cc:407] Connection header received
0926 16:43:19.802991 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:19.803021 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:19.803526 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:19.804042 (+ 516us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:19.806598 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:19.808265 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:19.810450 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:19.811305 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:19.811492 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:19.817932 (+ 6440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:19.817944 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:19.817967 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:19.818021 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:19.822563 (+ 4542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:19.823613 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:19.823625 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:19.823637 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:19.823902 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:19.824448 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:19.824456 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:19.824465 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:19.825373 (+ 908us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:19.825758 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:19.826247 (+ 489us) server_negotiation.cc:299] Negotiation successful
0926 16:43:19.826871 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1163,"thread_start_us":780,"threads_started":1}
I20260926 16:43:19.833812 11344 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17ee74c6a2a24bdcabd99741ead3b6fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ee74c6a2a24bdcabd99741ead3b6fc" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 33509 } }
I20260926 16:43:19.834322 11344 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17ee74c6a2a24bdcabd99741ead3b6fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:19.834458 11344 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17ee74c6a2a24bdcabd99741ead3b6fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17ee74c6a2a24bdcabd99741ead3b6fc, State: Initialized, Role: FOLLOWER
I20260926 16:43:19.835364 11344 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17ee74c6a2a24bdcabd99741ead3b6fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ee74c6a2a24bdcabd99741ead3b6fc" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 33509 } }
I20260926 16:43:19.835755 11344 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17ee74c6a2a24bdcabd99741ead3b6fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:43:19.835950 11344 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17ee74c6a2a24bdcabd99741ead3b6fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:43:19.836246 11344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17ee74c6a2a24bdcabd99741ead3b6fc [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:19.839473 11344 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17ee74c6a2a24bdcabd99741ead3b6fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ee74c6a2a24bdcabd99741ead3b6fc" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 33509 } }
I20260926 16:43:19.840430 11344 leader_election.cc:302] T 00000000000000000000000000000000 P 17ee74c6a2a24bdcabd99741ead3b6fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17ee74c6a2a24bdcabd99741ead3b6fc; no voters:
I20260926 16:43:19.841346 11344 leader_election.cc:288] T 00000000000000000000000000000000 P 17ee74c6a2a24bdcabd99741ead3b6fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:43:19.841534 11349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17ee74c6a2a24bdcabd99741ead3b6fc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:19.842463 11349 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17ee74c6a2a24bdcabd99741ead3b6fc [term 1 LEADER]: Becoming Leader. State: Replica: 17ee74c6a2a24bdcabd99741ead3b6fc, State: Running, Role: LEADER
I20260926 16:43:19.843207 11349 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17ee74c6a2a24bdcabd99741ead3b6fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ee74c6a2a24bdcabd99741ead3b6fc" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 33509 } }
I20260926 16:43:19.844247 11344 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17ee74c6a2a24bdcabd99741ead3b6fc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:43:19.847663 11351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17ee74c6a2a24bdcabd99741ead3b6fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17ee74c6a2a24bdcabd99741ead3b6fc. Latest consensus state: current_term: 1 leader_uuid: "17ee74c6a2a24bdcabd99741ead3b6fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ee74c6a2a24bdcabd99741ead3b6fc" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 33509 } } }
I20260926 16:43:19.848031 11350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17ee74c6a2a24bdcabd99741ead3b6fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17ee74c6a2a24bdcabd99741ead3b6fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17ee74c6a2a24bdcabd99741ead3b6fc" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 33509 } } }
I20260926 16:43:19.848670 11351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17ee74c6a2a24bdcabd99741ead3b6fc [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:19.848690 11350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17ee74c6a2a24bdcabd99741ead3b6fc [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:19.851747 11355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:43:19.857854 11355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:43:19.870332 11355 catalog_manager.cc:1347] Generated new cluster ID: 0884a9f125bd47a38e6096540ada8c6d
I20260926 16:43:19.870492 11355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:43:19.898247 11355 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:43:19.900358 11355 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:43:19.908638 11355 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17ee74c6a2a24bdcabd99741ead3b6fc: Generated new TSK 0
I20260926 16:43:19.909839 11355 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
I20260926 16:43:19.968680 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:0
--local_ip_for_outbound_sockets=127.9.238.193
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:33509
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:46317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:20.396488 11370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:20.396982 11370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:20.397146 11370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:20.405499 11370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:20.405710 11370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:20.405887 11370 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:43:20.406042 11370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:43:20.417672 11370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:46317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.238.254:33509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:20.419610 11370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:20.421351 11370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:20.438593 11378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:20.438642 11376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:20.438526 11375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:20.439623 11370 server_base.cc:1034] running on GCE node
I20260926 16:43:20.440536 11370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:20.441890 11370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:20.443298 11370 hybrid_clock.cc:648] HybridClock initialized: now 1790441000443162 us; error 69 us; skew 500 ppm
I20260926 16:43:20.443795 11370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:20 dist-test-slave-wt2j krb5kdc[11254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441000, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:20.448035 11370 init.cc:377] Logged in from keytab as kudu/127.9.238.193@KRBTEST.COM (short username kudu)
I20260926 16:43:20.450141 11370 webserver.cc:460] Webserver started at http://127.9.238.193:44633/ using document root <none> and password file <none>
I20260926 16:43:20.451117 11370 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:20.451225 11370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:20.451751 11370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:20.455897 11370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-0/data/instance:
uuid: "07c4e314672c437a95c60566c0d262cf"
format_stamp: "Formatted at 2026-09-26 16:43:20 on dist-test-slave-wt2j"
server_key: "490d52419e0226e5c3ef99b851e625ff"
server_key_iv: "794dfd78ecafb4fa774ebde5de1c6bf5"
server_key_version: "encryptionkey@0"
I20260926 16:43:20.456983 11370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance:
uuid: "07c4e314672c437a95c60566c0d262cf"
format_stamp: "Formatted at 2026-09-26 16:43:20 on dist-test-slave-wt2j"
server_key: "490d52419e0226e5c3ef99b851e625ff"
server_key_iv: "794dfd78ecafb4fa774ebde5de1c6bf5"
server_key_version: "encryptionkey@0"
I20260926 16:43:20.464344 11370 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260926 16:43:20.469331 11385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:20.471519 11370 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 16:43:20.471861 11370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "07c4e314672c437a95c60566c0d262cf"
format_stamp: "Formatted at 2026-09-26 16:43:20 on dist-test-slave-wt2j"
server_key: "490d52419e0226e5c3ef99b851e625ff"
server_key_iv: "794dfd78ecafb4fa774ebde5de1c6bf5"
server_key_version: "encryptionkey@0"
I20260926 16:43:20.472182 11370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:20.490013 11370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:20.496125 11370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:20.496515 11370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:20.500412 11370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:20.500564 11370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:20.500715 11370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:20.500871 11370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:43:20 dist-test-slave-wt2j krb5kdc[11254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441000, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:20.528707 11394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:20.501468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:41045 (local address 127.9.238.254:33509)
0926 16:43:20.502018 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:20.502024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:20.503897 (+ 1873us) server_negotiation.cc:407] Connection header received
0926 16:43:20.506565 (+ 2668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:20.506571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:20.506689 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:20.506815 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:20.509651 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:20.511094 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:20.513993 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:20.514746 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:20.514909 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:20.520032 (+ 5123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:20.520038 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:20.520041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:20.520077 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:20.523598 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:20.525092 (+ 1494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:20.525100 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:20.525104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:20.525292 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:20.526217 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:20.526224 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:20.526229 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:20.526599 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:20.526782 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:20.528078 (+ 1296us) server_negotiation.cc:299] Negotiation successful
0926 16:43:20.528318 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":293,"threads_started":1}
I20260926 16:43:20.529907 11395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:20.502462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:33509 (local address 127.9.238.193:41045)
0926 16:43:20.503673 (+ 1211us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:20.503750 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:20.506202 (+ 2452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:20.507146 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:20.507167 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:20.508193 (+ 1026us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:20.509327 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:20.509368 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:20.511417 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:20.511425 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:20.513691 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:20.513705 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:20.515097 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:20.515108 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:20.515327 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:20.516172 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:20.516198 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:20.519689 (+ 3491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:20.523805 (+ 4116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:20.523817 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:20.523848 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:20.524849 (+ 1001us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:20.525438 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:20.525446 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:20.525451 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:20.525970 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:20.526880 (+ 910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:20.526894 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:20.527529 (+ 635us) client_negotiation.cc:769] Sending connection context
0926 16:43:20.527970 (+ 441us) client_negotiation.cc:240] Negotiation successful
0926 16:43:20.528604 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"thread_start_us":485,"threads_started":1}
I20260926 16:43:20.557204 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:20.537907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:33509 (local address 127.9.238.193:56187)
0926 16:43:20.539197 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:20.539224 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:20.539358 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:20.539897 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:20.539900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:20.540431 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:20.540736 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:20.540745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:20.542669 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:20.542673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:20.543849 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:20.543861 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:20.545289 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:20.545304 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:20.545691 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:20.546597 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:20.546628 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:20.548336 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:20.553522 (+ 5186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:20.553527 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:20.553530 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:20.554145 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:20.554700 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:20.554708 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:20.554715 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:20.554914 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:20.555813 (+ 899us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:20.555817 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:20.556086 (+ 269us) client_negotiation.cc:769] Sending connection context
0926 16:43:20.556283 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 16:43:20.556802 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112,"thread_start_us":923,"threads_started":1}
I20260926 16:43:20.557754 11394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:20.537395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:56187 (local address 127.9.238.254:33509)
0926 16:43:20.537692 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:20.537703 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:20.539350 (+ 1647us) server_negotiation.cc:407] Connection header received
0926 16:43:20.539536 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:20.539576 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:20.539720 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:20.539876 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:20.540910 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:20.542454 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:20.544033 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:20.544762 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:20.544987 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:20.548629 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:20.548635 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:20.548640 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:20.548685 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:20.553306 (+ 4621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:20.554319 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:20.554325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:20.554329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:20.554535 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:20.555058 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:20.555065 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:20.555069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:20.555576 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:20.555761 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:20.556830 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0926 16:43:20.557174 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 16:43:20.565551 11370 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:42155
I20260926 16:43:20.565574 11505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:42155 every 8 connection(s)
I20260926 16:43:20.567309 11370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:43:20.569999 11506 heartbeater.cc:347] Connected to a master server at 127.9.238.254:33509
I20260926 16:43:20.570261 11506 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:20.570966 11506 heartbeater.cc:511] Master 127.9.238.254:33509 requested a full tablet report, sending...
I20260926 16:43:20.573240 11299 ts_manager.cc:194] Registered new tserver with Master: 07c4e314672c437a95c60566c0d262cf (127.9.238.193:42155)
I20260926 16:43:20.574719 11299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.193@KRBTEST.COM'} at 127.9.238.193:41045
I20260926 16:43:20.576500 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 11370
I20260926 16:43:20.576866 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance
I20260926 16:43:20.577751 10171 external_mini_cluster.cc:1428] Setting key 6327786bb4280ccfe9c5b3927bcc0fd5
WARNING: no policy specified for kudu/127.9.238.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
I20260926 16:43:20.628355 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:0
--local_ip_for_outbound_sockets=127.9.238.194
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:33509
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:46317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:21.045862 11510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:21.046296 11510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:21.046506 11510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:21.055869 11510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:21.056043 11510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:21.056192 11510 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:43:21.056358 11510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:43:21.068346 11510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:46317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.238.254:33509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:21.070268 11510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:21.072029 11510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:21.089545 11518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:21.089589 11516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:21.089864 11515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:21.090646 11510 server_base.cc:1034] running on GCE node
I20260926 16:43:21.091485 11510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:21.092710 11510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:21.093994 11510 hybrid_clock.cc:648] HybridClock initialized: now 1790441001093915 us; error 48 us; skew 500 ppm
I20260926 16:43:21.094421 11510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:21 dist-test-slave-wt2j krb5kdc[11254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441001, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:21.098388 11510 init.cc:377] Logged in from keytab as kudu/127.9.238.194@KRBTEST.COM (short username kudu)
I20260926 16:43:21.100566 11510 webserver.cc:460] Webserver started at http://127.9.238.194:46265/ using document root <none> and password file <none>
I20260926 16:43:21.101419 11510 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:21.101562 11510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:21.101917 11510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:21.105247 11510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-1/data/instance:
uuid: "98cdcd35007d45ebbbc6249314867b54"
format_stamp: "Formatted at 2026-09-26 16:43:21 on dist-test-slave-wt2j"
server_key: "de6c85d916645fd2e828a948aacb66f4"
server_key_iv: "5a1054ba8f8af45d3371e14404b6fcd2"
server_key_version: "encryptionkey@0"
I20260926 16:43:21.106122 11510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance:
uuid: "98cdcd35007d45ebbbc6249314867b54"
format_stamp: "Formatted at 2026-09-26 16:43:21 on dist-test-slave-wt2j"
server_key: "de6c85d916645fd2e828a948aacb66f4"
server_key_iv: "5a1054ba8f8af45d3371e14404b6fcd2"
server_key_version: "encryptionkey@0"
I20260926 16:43:21.113142 11510 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260926 16:43:21.119088 11525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:21.121680 11510 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260926 16:43:21.121906 11510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "98cdcd35007d45ebbbc6249314867b54"
format_stamp: "Formatted at 2026-09-26 16:43:21 on dist-test-slave-wt2j"
server_key: "de6c85d916645fd2e828a948aacb66f4"
server_key_iv: "5a1054ba8f8af45d3371e14404b6fcd2"
server_key_version: "encryptionkey@0"
I20260926 16:43:21.122164 11510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:21.137056 11510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:21.143158 11510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:21.143666 11510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:21.147408 11510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:21.147588 11510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:21.147769 11510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:21.147855 11510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:43:21 dist-test-slave-wt2j krb5kdc[11254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441001, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:21.175872 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:21.148349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:38885 (local address 127.9.238.254:33509)
0926 16:43:21.149012 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:21.149020 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:21.150414 (+ 1394us) server_negotiation.cc:407] Connection header received
0926 16:43:21.153075 (+ 2661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:21.153085 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:21.153248 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:21.153489 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:21.156383 (+ 2894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.157947 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.162823 (+ 4876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.163701 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.163883 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:21.167962 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:21.167967 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:21.167971 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:21.168049 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:21.172299 (+ 4250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:21.173356 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:21.173363 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:21.173366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:21.173518 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:21.173959 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:21.173964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:21.173967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:21.174266 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:21.174440 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:21.175205 (+ 765us) server_negotiation.cc:299] Negotiation successful
0926 16:43:21.175456 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":341,"threads_started":1}
I20260926 16:43:21.177356 11535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:21.149131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:33509 (local address 127.9.238.194:38885)
0926 16:43:21.150175 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:21.150229 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:21.152576 (+ 2347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:21.153755 (+ 1179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:21.153771 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:21.154897 (+ 1126us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:21.156061 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:21.156097 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.158241 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.158249 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:21.162455 (+ 4206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:21.162492 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.164034 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.164042 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:21.164302 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:21.165123 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:21.165149 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:21.167661 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:21.172556 (+ 4895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:21.172570 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:21.172595 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:21.173173 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:21.173639 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:21.173645 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:21.173649 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:21.173835 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:21.174394 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:21.174404 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:21.174820 (+ 416us) client_negotiation.cc:769] Sending connection context
0926 16:43:21.175095 (+ 275us) client_negotiation.cc:240] Negotiation successful
0926 16:43:21.175702 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":434,"threads_started":1}
I20260926 16:43:21.205307 11605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:21.187093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:33509 (local address 127.9.238.194:55829)
0926 16:43:21.187735 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:21.187770 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:21.187940 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:21.188584 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:21.188592 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:21.189422 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:21.190079 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:21.190097 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.192024 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.192031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:21.193268 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:21.193282 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.194449 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.194454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:21.194575 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:21.195403 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:21.195422 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:21.196572 (+ 1150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:21.201681 (+ 5109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:21.201689 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:21.201700 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:21.202482 (+ 782us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:21.203024 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:21.203031 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:21.203036 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:21.203248 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:21.204079 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:21.204084 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:21.204376 (+ 292us) client_negotiation.cc:769] Sending connection context
0926 16:43:21.204604 (+ 228us) client_negotiation.cc:240] Negotiation successful
0926 16:43:21.204958 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":576,"threads_started":1}
I20260926 16:43:21.205632 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:21.187431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:55829 (local address 127.9.238.254:33509)
0926 16:43:21.187653 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:21.187659 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:21.187909 (+ 250us) server_negotiation.cc:407] Connection header received
0926 16:43:21.188166 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:21.188175 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:21.188322 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:21.188476 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:21.190314 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.191793 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.193448 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.194237 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.194446 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:21.196867 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:21.196876 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:21.196883 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:21.196941 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:21.201457 (+ 4516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:21.202656 (+ 1199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:21.202664 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:21.202671 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:21.202878 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:21.203470 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:21.203476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:21.203478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:21.203949 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:21.204148 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:21.204778 (+ 630us) server_negotiation.cc:299] Negotiation successful
0926 16:43:21.205142 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 16:43:21.206708 11510 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:33185
I20260926 16:43:21.206744 11645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:33185 every 8 connection(s)
I20260926 16:43:21.208462 11510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:43:21.211071 11646 heartbeater.cc:347] Connected to a master server at 127.9.238.254:33509
I20260926 16:43:21.211439 11646 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:21.212301 11646 heartbeater.cc:511] Master 127.9.238.254:33509 requested a full tablet report, sending...
I20260926 16:43:21.213966 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 11510
I20260926 16:43:21.214175 11299 ts_manager.cc:194] Registered new tserver with Master: 98cdcd35007d45ebbbc6249314867b54 (127.9.238.194:33185)
I20260926 16:43:21.214303 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance
I20260926 16:43:21.214910 10171 external_mini_cluster.cc:1428] Setting key f446aff33c4e75f8c202836280e14cde
I20260926 16:43:21.215391 11299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.194@KRBTEST.COM'} at 127.9.238.194:38885
WARNING: no policy specified for kudu/127.9.238.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
I20260926 16:43:21.264634 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:0
--local_ip_for_outbound_sockets=127.9.238.195
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:33509
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:46317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:43:21.578210 11506 heartbeater.cc:503] Master 127.9.238.254:33509 was elected leader, sending a full tablet report...
W20260926 16:43:21.680331 11650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:21.680795 11650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:21.681025 11650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:21.689357 11650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:21.689507 11650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:21.689602 11650 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:43:21.689698 11650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:43:21.700987 11650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:46317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.238.254:33509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:21.702831 11650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:21.704633 11650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:21.721753 11655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:21.721727 11658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:21.721768 11656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:21.725587 11650 server_base.cc:1034] running on GCE node
I20260926 16:43:21.726614 11650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:21.728066 11650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:21.729343 11650 hybrid_clock.cc:648] HybridClock initialized: now 1790441001729261 us; error 50 us; skew 500 ppm
I20260926 16:43:21.729754 11650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:21 dist-test-slave-wt2j krb5kdc[11254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441001, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:21.734037 11650 init.cc:377] Logged in from keytab as kudu/127.9.238.195@KRBTEST.COM (short username kudu)
I20260926 16:43:21.736402 11650 webserver.cc:460] Webserver started at http://127.9.238.195:34157/ using document root <none> and password file <none>
I20260926 16:43:21.737392 11650 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:21.737555 11650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:21.737987 11650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:21.741961 11650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-2/data/instance:
uuid: "29744506ffd04e47905a7022ab1cfdb6"
format_stamp: "Formatted at 2026-09-26 16:43:21 on dist-test-slave-wt2j"
server_key: "6c57e801bc207599e539f54233bcf2dd"
server_key_iv: "72b9311ed72c44cf3b9dacdb183d5538"
server_key_version: "encryptionkey@0"
I20260926 16:43:21.742965 11650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance:
uuid: "29744506ffd04e47905a7022ab1cfdb6"
format_stamp: "Formatted at 2026-09-26 16:43:21 on dist-test-slave-wt2j"
server_key: "6c57e801bc207599e539f54233bcf2dd"
server_key_iv: "72b9311ed72c44cf3b9dacdb183d5538"
server_key_version: "encryptionkey@0"
I20260926 16:43:21.750181 11650 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260926 16:43:21.755280 11665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:21.757421 11650 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 16:43:21.757719 11650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "29744506ffd04e47905a7022ab1cfdb6"
format_stamp: "Formatted at 2026-09-26 16:43:21 on dist-test-slave-wt2j"
server_key: "6c57e801bc207599e539f54233bcf2dd"
server_key_iv: "72b9311ed72c44cf3b9dacdb183d5538"
server_key_version: "encryptionkey@0"
I20260926 16:43:21.758033 11650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:21.775803 11650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:21.782058 11650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:21.782415 11650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:21.786055 11650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:21.786212 11650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:21.786439 11650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:21.786533 11650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:43:21 dist-test-slave-wt2j krb5kdc[11254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441001, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:21.820437 11674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:21.787141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:37779 (local address 127.9.238.254:33509)
0926 16:43:21.790770 (+ 3629us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:21.790779 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:21.792344 (+ 1565us) server_negotiation.cc:407] Connection header received
0926 16:43:21.795587 (+ 3243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:21.795596 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:21.795699 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:21.795859 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:21.798716 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.800209 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.803244 (+ 3035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.804139 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.804342 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:21.809969 (+ 5627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:21.809981 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:21.809990 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:21.810068 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:21.814856 (+ 4788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:21.816432 (+ 1576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:21.816437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:21.816440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:21.816663 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:21.817673 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:21.817678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:21.817680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:21.818033 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:21.818164 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:21.819807 (+ 1643us) server_negotiation.cc:299] Negotiation successful
0926 16:43:21.820124 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3464,"thread_start_us":3313,"threads_started":1}
I20260926 16:43:21.823422 11675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:21.787724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:33509 (local address 127.9.238.195:37779)
0926 16:43:21.791640 (+ 3916us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:21.791712 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:21.795037 (+ 3325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:21.796241 (+ 1204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:21.796280 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:21.797297 (+ 1017us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:21.798386 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:21.798414 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.800475 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.800483 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:21.802963 (+ 2480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:21.802990 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.804530 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.804540 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:21.804924 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:21.806020 (+ 1096us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:21.806053 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:21.809498 (+ 3445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:21.815274 (+ 5776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:21.815288 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:21.815316 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:21.816200 (+ 884us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:21.816919 (+ 719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:21.816933 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:21.816940 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:21.817447 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:21.818221 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:21.818237 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:21.819108 (+ 871us) client_negotiation.cc:769] Sending connection context
0926 16:43:21.819671 (+ 563us) client_negotiation.cc:240] Negotiation successful
0926 16:43:21.820718 (+ 1047us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3520,"thread_start_us":3270,"threads_started":1}
I20260926 16:43:21.850329 11650 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:45795
I20260926 16:43:21.850348 11785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:45795 every 8 connection(s)
I20260926 16:43:21.851694 11674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:21.831812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:44013 (local address 127.9.238.254:33509)
0926 16:43:21.832224 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:21.832232 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:21.832538 (+ 306us) server_negotiation.cc:407] Connection header received
0926 16:43:21.832924 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:21.832935 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:21.833084 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:21.833265 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:21.834328 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.835792 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.838658 (+ 2866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.839196 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.839436 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:21.843279 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:21.843289 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:21.843296 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:21.843359 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:21.847744 (+ 4385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:21.849024 (+ 1280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:21.849031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:21.849035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:21.849232 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:21.849673 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:21.849682 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:21.849687 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:21.850183 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:21.850381 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:21.851097 (+ 716us) server_negotiation.cc:299] Negotiation successful
0926 16:43:21.851316 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248}
I20260926 16:43:21.852210 11752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:21.831820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:33509 (local address 127.9.238.195:44013)
0926 16:43:21.832429 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:21.832449 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:21.832655 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:21.833325 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:21.833328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:21.833831 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:21.834168 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:21.834178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.835979 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.835987 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:21.838440 (+ 2453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:21.838461 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.839587 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.839599 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:21.839915 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:21.840957 (+ 1042us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:21.840985 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:21.842917 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:21.848046 (+ 5129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:21.848052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:21.848055 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:21.848757 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:21.849394 (+ 637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:21.849398 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:21.849401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:21.849533 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:21.850408 (+ 875us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:21.850416 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:21.850755 (+ 339us) client_negotiation.cc:769] Sending connection context
0926 16:43:21.850979 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 16:43:21.851451 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":412,"threads_started":1}
I20260926 16:43:21.852916 11650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:43:21.855862 11786 heartbeater.cc:347] Connected to a master server at 127.9.238.254:33509
I20260926 16:43:21.856194 11786 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:21.857117 11786 heartbeater.cc:511] Master 127.9.238.254:33509 requested a full tablet report, sending...
I20260926 16:43:21.858322 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 11650
I20260926 16:43:21.858552 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance
I20260926 16:43:21.859074 10171 external_mini_cluster.cc:1428] Setting key 467dc22b960a5fb3cf13df681996d8f7
I20260926 16:43:21.859093 11299 ts_manager.cc:194] Registered new tserver with Master: 29744506ffd04e47905a7022ab1cfdb6 (127.9.238.195:45795)
I20260926 16:43:21.860283 11299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.195@KRBTEST.COM'} at 127.9.238.195:37779
I20260926 16:43:21.862854 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 16:43:21 dist-test-slave-wt2j krb5kdc[11254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441001, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 16:43:21 dist-test-slave-wt2j krb5kdc[11254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441001, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:21.917259 11674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:21.899817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43046 (local address 127.9.238.254:33509)
0926 16:43:21.900017 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:21.900021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:21.900489 (+ 468us) server_negotiation.cc:407] Connection header received
0926 16:43:21.900646 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:21.900678 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:21.900758 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:21.900884 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:21.902566 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.903426 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.905063 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.905536 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.905676 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:21.910421 (+ 4745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:21.910428 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:21.910431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:21.910479 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:21.913637 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:21.914526 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:21.914532 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:21.914535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:21.914715 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:21.915231 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:21.915235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:21.915237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:21.915619 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:21.915871 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:21.916472 (+ 601us) server_negotiation.cc:299] Negotiation successful
0926 16:43:21.916701 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 16:43:21.922974 11299 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:43:21.930449 11299 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:43:21.963709 11802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:21.949903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46844 (local address 127.9.238.195:45795)
0926 16:43:21.950876 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:21.950885 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:21.952074 (+ 1189us) server_negotiation.cc:407] Connection header received
0926 16:43:21.952485 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:21.952494 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:21.952809 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:21.952997 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:21.954542 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.956467 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.960426 (+ 3959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.961994 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.962085 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:21.962391 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:21.963073 (+ 682us) server_negotiation.cc:299] Negotiation successful
0926 16:43:21.963255 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":580,"threads_started":1}
I20260926 16:43:21.963687 11800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:21.949137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54510 (local address 127.9.238.194:33185)
0926 16:43:21.950060 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:21.950074 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:21.951281 (+ 1207us) server_negotiation.cc:407] Connection header received
0926 16:43:21.951878 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:21.951892 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:21.952280 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:21.952512 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:21.953851 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.955958 (+ 2107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.960340 (+ 4382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.961872 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.961997 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:21.962223 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:21.963048 (+ 825us) server_negotiation.cc:299] Negotiation successful
0926 16:43:21.963279 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":429,"threads_started":1}
I20260926 16:43:21.963727 11799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:21.948862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:33185 (local address 127.0.0.1:54510)
0926 16:43:21.951054 (+ 2192us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:21.951138 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:21.951507 (+ 369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:21.952988 (+ 1481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:21.952999 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:21.953047 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:21.953589 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:21.953606 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.956212 (+ 2606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.956222 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:21.960157 (+ 3935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:21.960170 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.962486 (+ 2316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.962493 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:21.962737 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:21.962796 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 16:43:21.962939 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 16:43:21.963175 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1340,"mutex_wait_us":872,"thread_start_us":1163,"threads_started":1}
I20260926 16:43:21.964967 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:21.949223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:45795 (local address 127.0.0.1:46844)
0926 16:43:21.951922 (+ 2699us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:21.951954 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:21.952256 (+ 302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:21.953549 (+ 1293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:21.953556 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:21.953594 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:21.954252 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:21.954277 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.956650 (+ 2373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.956658 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:21.960176 (+ 3518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:21.960194 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.962573 (+ 2379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.962578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:21.962850 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:21.962886 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 16:43:21.964413 (+ 1527us) client_negotiation.cc:240] Negotiation successful
0926 16:43:21.964569 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2553,"mutex_wait_us":86,"thread_start_us":9120,"threads_started":1}
I20260926 16:43:21.968546 11801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:21.951252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:42155 (local address 127.0.0.1:35608)
0926 16:43:21.958710 (+ 7458us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:21.958736 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:21.958842 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:21.960988 (+ 2146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:21.960995 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:21.961025 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:21.961411 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:21.961422 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.963257 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.963262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:21.965396 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:21.965414 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.967584 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.967592 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:21.967799 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:21.967840 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 16:43:21.967984 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 16:43:21.968149 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7291,"thread_start_us":1753,"threads_started":1}
I20260926 16:43:21.968551 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:21.951982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35608 (local address 127.9.238.193:42155)
0926 16:43:21.960072 (+ 8090us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:21.960082 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:21.960115 (+ 33us) server_negotiation.cc:407] Connection header received
0926 16:43:21.960224 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:21.960236 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:21.960559 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:21.960776 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:21.961653 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.963103 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.965616 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:21.967310 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:21.967488 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:21.967782 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:21.968037 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 16:43:21.968224 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7881,"thread_start_us":7759,"threads_started":1}
I20260926 16:43:21.968948 11576 tablet_service.cc:1467] Processing CreateTablet for tablet 985ba9e105124f6395707fe6b804f2dc (DEFAULT_TABLE table=test-table [id=06d1281e706a4b41990520082bcf230b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:21.968961 11716 tablet_service.cc:1467] Processing CreateTablet for tablet 985ba9e105124f6395707fe6b804f2dc (DEFAULT_TABLE table=test-table [id=06d1281e706a4b41990520082bcf230b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:21.970597 11716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 985ba9e105124f6395707fe6b804f2dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:21.970598 11576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 985ba9e105124f6395707fe6b804f2dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:21.976065 11436 tablet_service.cc:1467] Processing CreateTablet for tablet 985ba9e105124f6395707fe6b804f2dc (DEFAULT_TABLE table=test-table [id=06d1281e706a4b41990520082bcf230b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:21.977679 11436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 985ba9e105124f6395707fe6b804f2dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:21.986534 11805 tablet_bootstrap.cc:492] T 985ba9e105124f6395707fe6b804f2dc P 98cdcd35007d45ebbbc6249314867b54: Bootstrap starting.
I20260926 16:43:21.989944 11806 tablet_bootstrap.cc:492] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6: Bootstrap starting.
I20260926 16:43:21.992997 11807 tablet_bootstrap.cc:492] T 985ba9e105124f6395707fe6b804f2dc P 07c4e314672c437a95c60566c0d262cf: Bootstrap starting.
I20260926 16:43:21.993254 11805 tablet_bootstrap.cc:654] T 985ba9e105124f6395707fe6b804f2dc P 98cdcd35007d45ebbbc6249314867b54: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:21.994177 11806 tablet_bootstrap.cc:654] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:21.995092 11805 log.cc:824] T 985ba9e105124f6395707fe6b804f2dc P 98cdcd35007d45ebbbc6249314867b54: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:21.995417 11806 log.cc:824] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:21.997973 11807 tablet_bootstrap.cc:654] T 985ba9e105124f6395707fe6b804f2dc P 07c4e314672c437a95c60566c0d262cf: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:21.999267 11806 tablet_bootstrap.cc:492] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6: No bootstrap required, opened a new log
I20260926 16:43:21.999265 11805 tablet_bootstrap.cc:492] T 985ba9e105124f6395707fe6b804f2dc P 98cdcd35007d45ebbbc6249314867b54: No bootstrap required, opened a new log
I20260926 16:43:21.999614 11806 ts_tablet_manager.cc:1397] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 16:43:21.999614 11805 ts_tablet_manager.cc:1397] T 985ba9e105124f6395707fe6b804f2dc P 98cdcd35007d45ebbbc6249314867b54: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260926 16:43:21.999778 11807 log.cc:824] T 985ba9e105124f6395707fe6b804f2dc P 07c4e314672c437a95c60566c0d262cf: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:22.011703 11805 raft_consensus.cc:348] T 985ba9e105124f6395707fe6b804f2dc P 98cdcd35007d45ebbbc6249314867b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } } peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } }
I20260926 16:43:22.011839 11806 raft_consensus.cc:348] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } } peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } }
I20260926 16:43:22.012260 11806 raft_consensus.cc:374] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:22.012260 11805 raft_consensus.cc:374] T 985ba9e105124f6395707fe6b804f2dc P 98cdcd35007d45ebbbc6249314867b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:22.012363 11805 raft_consensus.cc:729] T 985ba9e105124f6395707fe6b804f2dc P 98cdcd35007d45ebbbc6249314867b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98cdcd35007d45ebbbc6249314867b54, State: Initialized, Role: FOLLOWER
I20260926 16:43:22.012363 11806 raft_consensus.cc:729] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29744506ffd04e47905a7022ab1cfdb6, State: Initialized, Role: FOLLOWER
I20260926 16:43:22.013278 11807 tablet_bootstrap.cc:492] T 985ba9e105124f6395707fe6b804f2dc P 07c4e314672c437a95c60566c0d262cf: No bootstrap required, opened a new log
I20260926 16:43:22.013181 11806 consensus_queue.cc:260] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } } peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } }
I20260926 16:43:22.013181 11805 consensus_queue.cc:260] T 985ba9e105124f6395707fe6b804f2dc P 98cdcd35007d45ebbbc6249314867b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } } peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } }
I20260926 16:43:22.013731 11807 ts_tablet_manager.cc:1397] T 985ba9e105124f6395707fe6b804f2dc P 07c4e314672c437a95c60566c0d262cf: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.004s
I20260926 16:43:22.016605 11786 heartbeater.cc:503] Master 127.9.238.254:33509 was elected leader, sending a full tablet report...
I20260926 16:43:22.017429 11806 ts_tablet_manager.cc:1428] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6: Time spent starting tablet: real 0.018s user 0.010s sys 0.004s
I20260926 16:43:22.017613 11646 heartbeater.cc:503] Master 127.9.238.254:33509 was elected leader, sending a full tablet report...
I20260926 16:43:22.017836 11805 ts_tablet_manager.cc:1428] T 985ba9e105124f6395707fe6b804f2dc P 98cdcd35007d45ebbbc6249314867b54: Time spent starting tablet: real 0.018s user 0.009s sys 0.007s
I20260926 16:43:22.028841 11807 raft_consensus.cc:348] T 985ba9e105124f6395707fe6b804f2dc P 07c4e314672c437a95c60566c0d262cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } } peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } }
I20260926 16:43:22.029414 11807 raft_consensus.cc:374] T 985ba9e105124f6395707fe6b804f2dc P 07c4e314672c437a95c60566c0d262cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:22.029542 11807 raft_consensus.cc:729] T 985ba9e105124f6395707fe6b804f2dc P 07c4e314672c437a95c60566c0d262cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c4e314672c437a95c60566c0d262cf, State: Initialized, Role: FOLLOWER
I20260926 16:43:22.030346 11807 consensus_queue.cc:260] T 985ba9e105124f6395707fe6b804f2dc P 07c4e314672c437a95c60566c0d262cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } } peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } }
I20260926 16:43:22.032673 11807 ts_tablet_manager.cc:1428] T 985ba9e105124f6395707fe6b804f2dc P 07c4e314672c437a95c60566c0d262cf: Time spent starting tablet: real 0.019s user 0.013s sys 0.004s
I20260926 16:43:22.046541 11812 raft_consensus.cc:482] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:43:22.047027 11812 raft_consensus.cc:504] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } } peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } }
I20260926 16:43:22.049335 11812 leader_election.cc:288] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98cdcd35007d45ebbbc6249314867b54 (127.9.238.194:33185), 07c4e314672c437a95c60566c0d262cf (127.9.238.193:42155)
I20260926 16:43:22.056679 11800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:22.049855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:51951 (local address 127.9.238.194:33185)
0926 16:43:22.050135 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:22.050143 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:22.050377 (+ 234us) server_negotiation.cc:407] Connection header received
0926 16:43:22.050721 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:22.050727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:22.050815 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:22.050923 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:22.051486 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.052764 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.054839 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.055692 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.055890 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:22.055973 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:22.056244 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 16:43:22.056373 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260926 16:43:22.056684 11675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:22.049750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:33185 (local address 127.9.238.195:51951)
0926 16:43:22.050142 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:22.050176 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:22.050490 (+ 314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:22.050994 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:22.051002 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:22.051059 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:22.051313 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:22.051325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.052926 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.052931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:22.054627 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:22.054641 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.055860 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.055867 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:22.056076 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:22.056102 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 16:43:22.056225 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 16:43:22.056372 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260926 16:43:22.057672 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:22.050592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:57437 (local address 127.9.238.193:42155)
0926 16:43:22.050872 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:22.050879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:22.051209 (+ 330us) server_negotiation.cc:407] Connection header received
0926 16:43:22.051426 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:22.051433 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:22.051539 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:22.051761 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:22.052370 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.053397 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.055574 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.056537 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.056757 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:22.056870 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:22.057204 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 16:43:22.057357 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 16:43:22.057816 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:22.050264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:42155 (local address 127.9.238.195:57437)
0926 16:43:22.051031 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:22.051053 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:22.051222 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:22.051767 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:22.051771 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:22.051792 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:22.052158 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:22.052168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.053662 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.053667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:22.055346 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:22.055362 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.056819 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.056826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:22.057018 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:22.057047 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 16:43:22.057173 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 16:43:22.057352 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":6,"thread_start_us":385,"threads_started":1}
I20260926 16:43:22.058110 11600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985ba9e105124f6395707fe6b804f2dc" candidate_uuid: "29744506ffd04e47905a7022ab1cfdb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98cdcd35007d45ebbbc6249314867b54" is_pre_election: true
I20260926 16:43:22.058717 11600 raft_consensus.cc:2454] T 985ba9e105124f6395707fe6b804f2dc P 98cdcd35007d45ebbbc6249314867b54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29744506ffd04e47905a7022ab1cfdb6 in term 0.
I20260926 16:43:22.059000 11461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985ba9e105124f6395707fe6b804f2dc" candidate_uuid: "29744506ffd04e47905a7022ab1cfdb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c4e314672c437a95c60566c0d262cf" is_pre_election: true
I20260926 16:43:22.059445 11669 leader_election.cc:302] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29744506ffd04e47905a7022ab1cfdb6, 98cdcd35007d45ebbbc6249314867b54; no voters:
I20260926 16:43:22.059751 11461 raft_consensus.cc:2454] T 985ba9e105124f6395707fe6b804f2dc P 07c4e314672c437a95c60566c0d262cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29744506ffd04e47905a7022ab1cfdb6 in term 0.
I20260926 16:43:22.059979 11812 raft_consensus.cc:2781] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:43:22.060189 11812 raft_consensus.cc:482] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:43:22.060328 11812 raft_consensus.cc:3037] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:22.063318 11812 raft_consensus.cc:504] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } } peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } }
I20260926 16:43:22.064206 11812 leader_election.cc:288] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6 [CANDIDATE]: Term 1 election: Requested vote from peers 98cdcd35007d45ebbbc6249314867b54 (127.9.238.194:33185), 07c4e314672c437a95c60566c0d262cf (127.9.238.193:42155)
I20260926 16:43:22.064908 11600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985ba9e105124f6395707fe6b804f2dc" candidate_uuid: "29744506ffd04e47905a7022ab1cfdb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98cdcd35007d45ebbbc6249314867b54"
I20260926 16:43:22.065001 11461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "985ba9e105124f6395707fe6b804f2dc" candidate_uuid: "29744506ffd04e47905a7022ab1cfdb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c4e314672c437a95c60566c0d262cf"
I20260926 16:43:22.065171 11600 raft_consensus.cc:3037] T 985ba9e105124f6395707fe6b804f2dc P 98cdcd35007d45ebbbc6249314867b54 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:22.065434 11461 raft_consensus.cc:3037] T 985ba9e105124f6395707fe6b804f2dc P 07c4e314672c437a95c60566c0d262cf [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:22.067735 11600 raft_consensus.cc:2454] T 985ba9e105124f6395707fe6b804f2dc P 98cdcd35007d45ebbbc6249314867b54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29744506ffd04e47905a7022ab1cfdb6 in term 1.
I20260926 16:43:22.068289 11669 leader_election.cc:302] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29744506ffd04e47905a7022ab1cfdb6, 98cdcd35007d45ebbbc6249314867b54; no voters:
I20260926 16:43:22.068655 11812 raft_consensus.cc:2781] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:22.068708 11461 raft_consensus.cc:2454] T 985ba9e105124f6395707fe6b804f2dc P 07c4e314672c437a95c60566c0d262cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29744506ffd04e47905a7022ab1cfdb6 in term 1.
I20260926 16:43:22.069314 11812 raft_consensus.cc:686] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6 [term 1 LEADER]: Becoming Leader. State: Replica: 29744506ffd04e47905a7022ab1cfdb6, State: Running, Role: LEADER
I20260926 16:43:22.069978 11812 consensus_queue.cc:237] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } } peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } }
W20260926 16:43:22.070773 11507 tablet.cc:2369] T 985ba9e105124f6395707fe6b804f2dc P 07c4e314672c437a95c60566c0d262cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:43:22.076185 11298 catalog_manager.cc:5495] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29744506ffd04e47905a7022ab1cfdb6 (127.9.238.195). New cstate: current_term: 1 leader_uuid: "29744506ffd04e47905a7022ab1cfdb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } health_report { overall_health: UNKNOWN } } }
W20260926 16:43:22.105118 11787 tablet.cc:2369] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:43:22.109061 11802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:22.101482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46854 (local address 127.9.238.195:45795)
0926 16:43:22.101746 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:22.101752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:22.101947 (+ 195us) server_negotiation.cc:407] Connection header received
0926 16:43:22.102334 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:22.102338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:22.102418 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:22.102565 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:43:22.102953 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.103737 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.105421 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.106002 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.106167 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:22.106733 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:43:22.107865 (+ 1132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:43:22.108039 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:22.108353 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 16:43:22.108753 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260926 16:43:22.121279 11461 raft_consensus.cc:1261] T 985ba9e105124f6395707fe6b804f2dc P 07c4e314672c437a95c60566c0d262cf [term 1 FOLLOWER]: Refusing update from remote peer 29744506ffd04e47905a7022ab1cfdb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:43:22.121363 11600 raft_consensus.cc:1261] T 985ba9e105124f6395707fe6b804f2dc P 98cdcd35007d45ebbbc6249314867b54 [term 1 FOLLOWER]: Refusing update from remote peer 29744506ffd04e47905a7022ab1cfdb6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:43:22.122680 11815 consensus_queue.cc:1035] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:43:22.123099 11812 consensus_queue.cc:1035] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:43:22.143451 11819 mvcc.cc:205] Tried to move back new op lower bound from 7333646344672796672 to 7333646344484163584. Current Snapshot: MvccSnapshot[applied={T|T < 7333646344672796672}]
I20260926 16:43:22.164390 11297 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:43:22.164786 11297 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:43:22.169106 11297 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17ee74c6a2a24bdcabd99741ead3b6fc: Sending DeleteTablet for 3 replicas of tablet 985ba9e105124f6395707fe6b804f2dc
I20260926 16:43:22.170466 11576 tablet_service.cc:1514] Processing DeleteTablet for tablet 985ba9e105124f6395707fe6b804f2dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:22 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:54510
I20260926 16:43:22.170531 11436 tablet_service.cc:1514] Processing DeleteTablet for tablet 985ba9e105124f6395707fe6b804f2dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:22 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:35608
I20260926 16:43:22.171087 11716 tablet_service.cc:1514] Processing DeleteTablet for tablet 985ba9e105124f6395707fe6b804f2dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:22 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:46844
I20260926 16:43:22.177897 11297 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:43:22.180325 11297 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:43:22.181854 11830 tablet_replica.cc:330] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6: stopping tablet replica
I20260926 16:43:22.182183 11828 tablet_replica.cc:330] T 985ba9e105124f6395707fe6b804f2dc P 07c4e314672c437a95c60566c0d262cf: stopping tablet replica
I20260926 16:43:22.182570 11830 raft_consensus.cc:2229] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:43:22.182608 11828 raft_consensus.cc:2229] T 985ba9e105124f6395707fe6b804f2dc P 07c4e314672c437a95c60566c0d262cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:43:22.183264 11828 raft_consensus.cc:2258] T 985ba9e105124f6395707fe6b804f2dc P 07c4e314672c437a95c60566c0d262cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:43:22.183390 11830 raft_consensus.cc:2258] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:43:22.184353 11829 tablet_replica.cc:330] T 985ba9e105124f6395707fe6b804f2dc P 98cdcd35007d45ebbbc6249314867b54: stopping tablet replica
I20260926 16:43:22.184888 11829 raft_consensus.cc:2229] T 985ba9e105124f6395707fe6b804f2dc P 98cdcd35007d45ebbbc6249314867b54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:43:22.185434 11829 raft_consensus.cc:2258] T 985ba9e105124f6395707fe6b804f2dc P 98cdcd35007d45ebbbc6249314867b54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:43:22.188000 11829 ts_tablet_manager.cc:1905] T 985ba9e105124f6395707fe6b804f2dc P 98cdcd35007d45ebbbc6249314867b54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:43:22.191179 11436 tablet_service.cc:1467] Processing CreateTablet for tablet 3d33ad0ad6e146b092473f0ba7999c33 (DEFAULT_TABLE table=test-table [id=f8fc35aec0db4729bd6105ba16a08399]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:22.191319 11716 tablet_service.cc:1467] Processing CreateTablet for tablet 3d33ad0ad6e146b092473f0ba7999c33 (DEFAULT_TABLE table=test-table [id=f8fc35aec0db4729bd6105ba16a08399]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:22.191794 11436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d33ad0ad6e146b092473f0ba7999c33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:22.191989 11716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d33ad0ad6e146b092473f0ba7999c33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:22.192497 11828 ts_tablet_manager.cc:1905] T 985ba9e105124f6395707fe6b804f2dc P 07c4e314672c437a95c60566c0d262cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:43:22.194864 11830 ts_tablet_manager.cc:1905] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:43:22.198997 11807 tablet_bootstrap.cc:492] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf: Bootstrap starting.
I20260926 16:43:22.199496 11576 tablet_service.cc:1467] Processing CreateTablet for tablet 3d33ad0ad6e146b092473f0ba7999c33 (DEFAULT_TABLE table=test-table [id=f8fc35aec0db4729bd6105ba16a08399]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:22.200054 11576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d33ad0ad6e146b092473f0ba7999c33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:22.201155 11807 tablet_bootstrap.cc:654] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:22.202739 11806 tablet_bootstrap.cc:492] T 3d33ad0ad6e146b092473f0ba7999c33 P 29744506ffd04e47905a7022ab1cfdb6: Bootstrap starting.
I20260926 16:43:22.204820 11806 tablet_bootstrap.cc:654] T 3d33ad0ad6e146b092473f0ba7999c33 P 29744506ffd04e47905a7022ab1cfdb6: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:22.208093 11805 tablet_bootstrap.cc:492] T 3d33ad0ad6e146b092473f0ba7999c33 P 98cdcd35007d45ebbbc6249314867b54: Bootstrap starting.
I20260926 16:43:22.210001 11805 tablet_bootstrap.cc:654] T 3d33ad0ad6e146b092473f0ba7999c33 P 98cdcd35007d45ebbbc6249314867b54: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:22.214152 11828 ts_tablet_manager.cc:1918] T 985ba9e105124f6395707fe6b804f2dc P 07c4e314672c437a95c60566c0d262cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:43:22.214344 11828 log.cc:1196] T 985ba9e105124f6395707fe6b804f2dc P 07c4e314672c437a95c60566c0d262cf: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-0/wal/wals/985ba9e105124f6395707fe6b804f2dc
I20260926 16:43:22.215056 11828 ts_tablet_manager.cc:1939] T 985ba9e105124f6395707fe6b804f2dc P 07c4e314672c437a95c60566c0d262cf: Deleting consensus metadata
I20260926 16:43:22.218747 11829 ts_tablet_manager.cc:1918] T 985ba9e105124f6395707fe6b804f2dc P 98cdcd35007d45ebbbc6249314867b54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:43:22.221505 11286 catalog_manager.cc:4841] TS 07c4e314672c437a95c60566c0d262cf (127.9.238.193:42155): tablet 985ba9e105124f6395707fe6b804f2dc (table test-table [id=06d1281e706a4b41990520082bcf230b]) successfully deleted
I20260926 16:43:22.222577 11807 tablet_bootstrap.cc:492] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf: No bootstrap required, opened a new log
I20260926 16:43:22.222755 11807 ts_tablet_manager.cc:1397] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.007s
I20260926 16:43:22.222890 11805 tablet_bootstrap.cc:492] T 3d33ad0ad6e146b092473f0ba7999c33 P 98cdcd35007d45ebbbc6249314867b54: No bootstrap required, opened a new log
I20260926 16:43:22.222941 11829 log.cc:1196] T 985ba9e105124f6395707fe6b804f2dc P 98cdcd35007d45ebbbc6249314867b54: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-1/wal/wals/985ba9e105124f6395707fe6b804f2dc
I20260926 16:43:22.223080 11805 ts_tablet_manager.cc:1397] T 3d33ad0ad6e146b092473f0ba7999c33 P 98cdcd35007d45ebbbc6249314867b54: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20260926 16:43:22.224040 11829 ts_tablet_manager.cc:1939] T 985ba9e105124f6395707fe6b804f2dc P 98cdcd35007d45ebbbc6249314867b54: Deleting consensus metadata
I20260926 16:43:22.224413 11830 ts_tablet_manager.cc:1918] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:43:22.224647 11830 log.cc:1196] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-2/wal/wals/985ba9e105124f6395707fe6b804f2dc
I20260926 16:43:22.225024 11807 raft_consensus.cc:348] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } }
I20260926 16:43:22.225237 11807 raft_consensus.cc:374] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:22.225258 11830 ts_tablet_manager.cc:1939] T 985ba9e105124f6395707fe6b804f2dc P 29744506ffd04e47905a7022ab1cfdb6: Deleting consensus metadata
I20260926 16:43:22.225329 11807 raft_consensus.cc:729] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c4e314672c437a95c60566c0d262cf, State: Initialized, Role: FOLLOWER
I20260926 16:43:22.225654 11807 consensus_queue.cc:260] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } }
I20260926 16:43:22.226215 11807 ts_tablet_manager.cc:1428] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260926 16:43:22.224210 11805 raft_consensus.cc:348] T 3d33ad0ad6e146b092473f0ba7999c33 P 98cdcd35007d45ebbbc6249314867b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } }
I20260926 16:43:22.226722 11805 raft_consensus.cc:374] T 3d33ad0ad6e146b092473f0ba7999c33 P 98cdcd35007d45ebbbc6249314867b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:22.226848 11805 raft_consensus.cc:729] T 3d33ad0ad6e146b092473f0ba7999c33 P 98cdcd35007d45ebbbc6249314867b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98cdcd35007d45ebbbc6249314867b54, State: Initialized, Role: FOLLOWER
I20260926 16:43:22.227180 11805 consensus_queue.cc:260] T 3d33ad0ad6e146b092473f0ba7999c33 P 98cdcd35007d45ebbbc6249314867b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } }
I20260926 16:43:22.228257 11805 ts_tablet_manager.cc:1428] T 3d33ad0ad6e146b092473f0ba7999c33 P 98cdcd35007d45ebbbc6249314867b54: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260926 16:43:22.228647 11284 catalog_manager.cc:4841] TS 29744506ffd04e47905a7022ab1cfdb6 (127.9.238.195:45795): tablet 985ba9e105124f6395707fe6b804f2dc (table test-table [id=06d1281e706a4b41990520082bcf230b]) successfully deleted
I20260926 16:43:22.230631 11286 catalog_manager.cc:4841] TS 98cdcd35007d45ebbbc6249314867b54 (127.9.238.194:33185): tablet 985ba9e105124f6395707fe6b804f2dc (table test-table [id=06d1281e706a4b41990520082bcf230b]) successfully deleted
I20260926 16:43:22.230821 11806 tablet_bootstrap.cc:492] T 3d33ad0ad6e146b092473f0ba7999c33 P 29744506ffd04e47905a7022ab1cfdb6: No bootstrap required, opened a new log
I20260926 16:43:22.231040 11806 ts_tablet_manager.cc:1397] T 3d33ad0ad6e146b092473f0ba7999c33 P 29744506ffd04e47905a7022ab1cfdb6: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.005s
I20260926 16:43:22.232163 11806 raft_consensus.cc:348] T 3d33ad0ad6e146b092473f0ba7999c33 P 29744506ffd04e47905a7022ab1cfdb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } }
I20260926 16:43:22.232519 11806 raft_consensus.cc:374] T 3d33ad0ad6e146b092473f0ba7999c33 P 29744506ffd04e47905a7022ab1cfdb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:22.232622 11806 raft_consensus.cc:729] T 3d33ad0ad6e146b092473f0ba7999c33 P 29744506ffd04e47905a7022ab1cfdb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29744506ffd04e47905a7022ab1cfdb6, State: Initialized, Role: FOLLOWER
I20260926 16:43:22.233021 11806 consensus_queue.cc:260] T 3d33ad0ad6e146b092473f0ba7999c33 P 29744506ffd04e47905a7022ab1cfdb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } }
I20260926 16:43:22.234637 11806 ts_tablet_manager.cc:1428] T 3d33ad0ad6e146b092473f0ba7999c33 P 29744506ffd04e47905a7022ab1cfdb6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 16:43:22.272231 11813 raft_consensus.cc:482] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:43:22.272547 11813 raft_consensus.cc:504] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } }
I20260926 16:43:22.274537 11813 leader_election.cc:288] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98cdcd35007d45ebbbc6249314867b54 (127.9.238.194:33185), 29744506ffd04e47905a7022ab1cfdb6 (127.9.238.195:45795)
I20260926 16:43:22.285331 11802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:22.275683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:57169 (local address 127.9.238.195:45795)
0926 16:43:22.275951 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:22.275961 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:22.277362 (+ 1401us) server_negotiation.cc:407] Connection header received
0926 16:43:22.277880 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:22.277885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:22.278122 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:22.278193 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:22.279298 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.280243 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.283203 (+ 2960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.284221 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.284326 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:22.284385 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:22.284832 (+ 447us) server_negotiation.cc:299] Negotiation successful
0926 16:43:22.284970 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 16:43:22.285403 11834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:22.275142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:45795 (local address 127.9.238.193:57169)
0926 16:43:22.277233 (+ 2091us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:22.277262 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:22.277642 (+ 380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:22.278439 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:22.278442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:22.278463 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:22.279066 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:22.279080 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.280583 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.280586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:22.282959 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:22.282974 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.284537 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.284541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:22.284659 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:22.284683 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 16:43:22.284792 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 16:43:22.284938 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1889,"mutex_wait_us":50,"thread_start_us":1681,"threads_started":1}
I20260926 16:43:22.286669 11800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:22.276092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:38697 (local address 127.9.238.194:33185)
0926 16:43:22.276359 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:22.276370 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:22.278056 (+ 1686us) server_negotiation.cc:407] Connection header received
0926 16:43:22.278298 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:22.278304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:22.278415 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:22.278572 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:22.279356 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.280768 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.283272 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.284686 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.284882 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:22.285450 (+ 568us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:22.286154 (+ 704us) server_negotiation.cc:299] Negotiation successful
0926 16:43:22.286356 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 16:43:22.286721 11835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:22.275152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:33185 (local address 127.9.238.193:38697)
0926 16:43:22.277916 (+ 2764us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:22.277934 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:22.278058 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:22.278600 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:22.278606 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:22.278630 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:22.279186 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:22.279199 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.280995 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.281001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:22.283103 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:22.283119 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.285641 (+ 2522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.285649 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:22.285904 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:22.285946 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 16:43:22.286066 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 16:43:22.286246 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2638,"mutex_wait_us":65,"thread_start_us":2104,"threads_started":1}
I20260926 16:43:22.286669 11736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d33ad0ad6e146b092473f0ba7999c33" candidate_uuid: "07c4e314672c437a95c60566c0d262cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29744506ffd04e47905a7022ab1cfdb6" is_pre_election: true
I20260926 16:43:22.287209 11736 raft_consensus.cc:2454] T 3d33ad0ad6e146b092473f0ba7999c33 P 29744506ffd04e47905a7022ab1cfdb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07c4e314672c437a95c60566c0d262cf in term 0.
I20260926 16:43:22.287627 11600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d33ad0ad6e146b092473f0ba7999c33" candidate_uuid: "07c4e314672c437a95c60566c0d262cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98cdcd35007d45ebbbc6249314867b54" is_pre_election: true
I20260926 16:43:22.287957 11600 raft_consensus.cc:2454] T 3d33ad0ad6e146b092473f0ba7999c33 P 98cdcd35007d45ebbbc6249314867b54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07c4e314672c437a95c60566c0d262cf in term 0.
I20260926 16:43:22.287911 11387 leader_election.cc:302] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c4e314672c437a95c60566c0d262cf, 29744506ffd04e47905a7022ab1cfdb6; no voters:
I20260926 16:43:22.288502 11813 raft_consensus.cc:2781] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:43:22.288652 11813 raft_consensus.cc:482] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:43:22.288757 11813 raft_consensus.cc:3037] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:22.292469 11813 raft_consensus.cc:504] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } }
I20260926 16:43:22.293361 11813 leader_election.cc:288] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf [CANDIDATE]: Term 1 election: Requested vote from peers 98cdcd35007d45ebbbc6249314867b54 (127.9.238.194:33185), 29744506ffd04e47905a7022ab1cfdb6 (127.9.238.195:45795)
I20260926 16:43:22.294188 11600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d33ad0ad6e146b092473f0ba7999c33" candidate_uuid: "07c4e314672c437a95c60566c0d262cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98cdcd35007d45ebbbc6249314867b54"
I20260926 16:43:22.294205 11736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d33ad0ad6e146b092473f0ba7999c33" candidate_uuid: "07c4e314672c437a95c60566c0d262cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29744506ffd04e47905a7022ab1cfdb6"
I20260926 16:43:22.294464 11736 raft_consensus.cc:3037] T 3d33ad0ad6e146b092473f0ba7999c33 P 29744506ffd04e47905a7022ab1cfdb6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:22.294472 11600 raft_consensus.cc:3037] T 3d33ad0ad6e146b092473f0ba7999c33 P 98cdcd35007d45ebbbc6249314867b54 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:22.297590 11736 raft_consensus.cc:2454] T 3d33ad0ad6e146b092473f0ba7999c33 P 29744506ffd04e47905a7022ab1cfdb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07c4e314672c437a95c60566c0d262cf in term 1.
I20260926 16:43:22.297909 11600 raft_consensus.cc:2454] T 3d33ad0ad6e146b092473f0ba7999c33 P 98cdcd35007d45ebbbc6249314867b54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07c4e314672c437a95c60566c0d262cf in term 1.
I20260926 16:43:22.298094 11387 leader_election.cc:302] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c4e314672c437a95c60566c0d262cf, 29744506ffd04e47905a7022ab1cfdb6; no voters:
I20260926 16:43:22.298529 11813 raft_consensus.cc:2781] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:22.299077 11813 raft_consensus.cc:686] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf [term 1 LEADER]: Becoming Leader. State: Replica: 07c4e314672c437a95c60566c0d262cf, State: Running, Role: LEADER
I20260926 16:43:22.299523 11813 consensus_queue.cc:237] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } }
I20260926 16:43:22.303057 11298 catalog_manager.cc:5495] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 07c4e314672c437a95c60566c0d262cf (127.9.238.193). New cstate: current_term: 1 leader_uuid: "07c4e314672c437a95c60566c0d262cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } health_report { overall_health: UNKNOWN } } }
Sep 26 16:43:22 dist-test-slave-wt2j krb5kdc[11254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790440999, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:22.366231 11674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:22.346585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43056 (local address 127.9.238.254:33509)
0926 16:43:22.346888 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:22.346898 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:22.347497 (+ 599us) server_negotiation.cc:407] Connection header received
0926 16:43:22.347764 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:22.347771 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:22.347884 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:22.348057 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:22.349534 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.350914 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.352749 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.353209 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.353367 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:22.357913 (+ 4546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:22.357918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:22.357921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:22.357950 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:22.361728 (+ 3778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:22.362909 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:22.362923 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:22.362929 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:22.363160 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:22.364167 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:22.364172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:22.364174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:22.364487 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:22.364623 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:22.365403 (+ 780us) server_negotiation.cc:299] Negotiation successful
0926 16:43:22.365876 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":222,"server-negotiator.queue_time_us":94}
I20260926 16:43:22.366426 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:22.346598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:33509 (local address 127.0.0.1:43056)
0926 16:43:22.347361 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:22.347380 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:22.347532 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:22.348133 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:22.348137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:22.348988 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:22.349285 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:22.349294 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.351158 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.351163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:22.352571 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:22.352587 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.353406 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.353412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:22.353598 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:22.354696 (+ 1098us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:22.354714 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:22.357680 (+ 2966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:22.361916 (+ 4236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:22.361936 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:22.361946 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:22.362715 (+ 769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:22.363493 (+ 778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:22.363505 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:22.363510 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:22.363919 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:22.364662 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:22.364674 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:22.365000 (+ 326us) client_negotiation.cc:769] Sending connection context
0926 16:43:22.365252 (+ 252us) client_negotiation.cc:240] Negotiation successful
0926 16:43:22.365709 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":419,"threads_started":1}
I20260926 16:43:22.374953 11298 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:43056:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 16:43:22.376362 11298 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:43:22.386036 11576 tablet_service.cc:1467] Processing CreateTablet for tablet 9aee2f8c06b94360a5e1efdcdeae342b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fe05640611544d77baea27dc8c0e8087]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:43:22.386036 11436 tablet_service.cc:1467] Processing CreateTablet for tablet 9aee2f8c06b94360a5e1efdcdeae342b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fe05640611544d77baea27dc8c0e8087]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:43:22.386755 11576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aee2f8c06b94360a5e1efdcdeae342b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:22.386744 11436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aee2f8c06b94360a5e1efdcdeae342b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:22.386662 11716 tablet_service.cc:1467] Processing CreateTablet for tablet 9aee2f8c06b94360a5e1efdcdeae342b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fe05640611544d77baea27dc8c0e8087]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:43:22.387523 11716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aee2f8c06b94360a5e1efdcdeae342b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:22.395819 11807 tablet_bootstrap.cc:492] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf: Bootstrap starting.
I20260926 16:43:22.396834 11805 tablet_bootstrap.cc:492] T 9aee2f8c06b94360a5e1efdcdeae342b P 98cdcd35007d45ebbbc6249314867b54: Bootstrap starting.
I20260926 16:43:22.397327 11806 tablet_bootstrap.cc:492] T 9aee2f8c06b94360a5e1efdcdeae342b P 29744506ffd04e47905a7022ab1cfdb6: Bootstrap starting.
I20260926 16:43:22.397989 11807 tablet_bootstrap.cc:654] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:22.398757 11805 tablet_bootstrap.cc:654] T 9aee2f8c06b94360a5e1efdcdeae342b P 98cdcd35007d45ebbbc6249314867b54: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:22.399470 11806 tablet_bootstrap.cc:654] T 9aee2f8c06b94360a5e1efdcdeae342b P 29744506ffd04e47905a7022ab1cfdb6: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:22.407375 11807 tablet_bootstrap.cc:492] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf: No bootstrap required, opened a new log
I20260926 16:43:22.407747 11807 ts_tablet_manager.cc:1397] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260926 16:43:22.408844 11807 raft_consensus.cc:348] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } }
I20260926 16:43:22.409149 11807 raft_consensus.cc:374] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:22.409243 11807 raft_consensus.cc:729] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c4e314672c437a95c60566c0d262cf, State: Initialized, Role: FOLLOWER
I20260926 16:43:22.409543 11807 consensus_queue.cc:260] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } }
I20260926 16:43:22.410000 11813 tablet_replica.cc:439] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } } }
I20260926 16:43:22.410209 11807 ts_tablet_manager.cc:1428] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 16:43:22.410380 11813 tablet_replica.cc:442] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:43:22.411602 11806 tablet_bootstrap.cc:492] T 9aee2f8c06b94360a5e1efdcdeae342b P 29744506ffd04e47905a7022ab1cfdb6: No bootstrap required, opened a new log
I20260926 16:43:22.411890 11806 ts_tablet_manager.cc:1397] T 9aee2f8c06b94360a5e1efdcdeae342b P 29744506ffd04e47905a7022ab1cfdb6: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260926 16:43:22.412962 11806 raft_consensus.cc:348] T 9aee2f8c06b94360a5e1efdcdeae342b P 29744506ffd04e47905a7022ab1cfdb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } }
I20260926 16:43:22.413307 11806 raft_consensus.cc:374] T 9aee2f8c06b94360a5e1efdcdeae342b P 29744506ffd04e47905a7022ab1cfdb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:22.413419 11806 raft_consensus.cc:729] T 9aee2f8c06b94360a5e1efdcdeae342b P 29744506ffd04e47905a7022ab1cfdb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29744506ffd04e47905a7022ab1cfdb6, State: Initialized, Role: FOLLOWER
I20260926 16:43:22.413597 11805 tablet_bootstrap.cc:492] T 9aee2f8c06b94360a5e1efdcdeae342b P 98cdcd35007d45ebbbc6249314867b54: No bootstrap required, opened a new log
I20260926 16:43:22.413825 11805 ts_tablet_manager.cc:1397] T 9aee2f8c06b94360a5e1efdcdeae342b P 98cdcd35007d45ebbbc6249314867b54: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.000s
I20260926 16:43:22.413751 11806 consensus_queue.cc:260] T 9aee2f8c06b94360a5e1efdcdeae342b P 29744506ffd04e47905a7022ab1cfdb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } }
I20260926 16:43:22.414343 11815 tablet_replica.cc:439] T 9aee2f8c06b94360a5e1efdcdeae342b P 29744506ffd04e47905a7022ab1cfdb6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } } }
I20260926 16:43:22.414737 11815 tablet_replica.cc:442] T 9aee2f8c06b94360a5e1efdcdeae342b P 29744506ffd04e47905a7022ab1cfdb6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:43:22.414711 11805 raft_consensus.cc:348] T 9aee2f8c06b94360a5e1efdcdeae342b P 98cdcd35007d45ebbbc6249314867b54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } }
I20260926 16:43:22.414978 11805 raft_consensus.cc:374] T 9aee2f8c06b94360a5e1efdcdeae342b P 98cdcd35007d45ebbbc6249314867b54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:22.415310 11805 raft_consensus.cc:729] T 9aee2f8c06b94360a5e1efdcdeae342b P 98cdcd35007d45ebbbc6249314867b54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98cdcd35007d45ebbbc6249314867b54, State: Initialized, Role: FOLLOWER
I20260926 16:43:22.415396 11806 ts_tablet_manager.cc:1428] T 9aee2f8c06b94360a5e1efdcdeae342b P 29744506ffd04e47905a7022ab1cfdb6: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 16:43:22.415722 11805 consensus_queue.cc:260] T 9aee2f8c06b94360a5e1efdcdeae342b P 98cdcd35007d45ebbbc6249314867b54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } }
I20260926 16:43:22.416273 11811 tablet_replica.cc:439] T 9aee2f8c06b94360a5e1efdcdeae342b P 98cdcd35007d45ebbbc6249314867b54: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } } }
I20260926 16:43:22.416677 11811 tablet_replica.cc:442] T 9aee2f8c06b94360a5e1efdcdeae342b P 98cdcd35007d45ebbbc6249314867b54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:43:22.418541 11805 ts_tablet_manager.cc:1428] T 9aee2f8c06b94360a5e1efdcdeae342b P 98cdcd35007d45ebbbc6249314867b54: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
W20260926 16:43:22.462949 11647 tablet.cc:2369] T 3d33ad0ad6e146b092473f0ba7999c33 P 98cdcd35007d45ebbbc6249314867b54: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:43:22.483843 11813 raft_consensus.cc:482] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:43:22.484179 11813 raft_consensus.cc:504] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } }
I20260926 16:43:22.484957 11813 leader_election.cc:288] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98cdcd35007d45ebbbc6249314867b54 (127.9.238.194:33185), 29744506ffd04e47905a7022ab1cfdb6 (127.9.238.195:45795)
I20260926 16:43:22.485622 11600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aee2f8c06b94360a5e1efdcdeae342b" candidate_uuid: "07c4e314672c437a95c60566c0d262cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98cdcd35007d45ebbbc6249314867b54" is_pre_election: true
I20260926 16:43:22.485731 11736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aee2f8c06b94360a5e1efdcdeae342b" candidate_uuid: "07c4e314672c437a95c60566c0d262cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29744506ffd04e47905a7022ab1cfdb6" is_pre_election: true
I20260926 16:43:22.485982 11600 raft_consensus.cc:2454] T 9aee2f8c06b94360a5e1efdcdeae342b P 98cdcd35007d45ebbbc6249314867b54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07c4e314672c437a95c60566c0d262cf in term 0.
I20260926 16:43:22.486004 11736 raft_consensus.cc:2454] T 9aee2f8c06b94360a5e1efdcdeae342b P 29744506ffd04e47905a7022ab1cfdb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07c4e314672c437a95c60566c0d262cf in term 0.
I20260926 16:43:22.486588 11389 leader_election.cc:302] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 07c4e314672c437a95c60566c0d262cf, 29744506ffd04e47905a7022ab1cfdb6, 98cdcd35007d45ebbbc6249314867b54; no voters:
I20260926 16:43:22.486966 11813 raft_consensus.cc:2781] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:43:22.487071 11813 raft_consensus.cc:482] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:43:22.487167 11813 raft_consensus.cc:3037] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:22.489337 11813 raft_consensus.cc:504] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } }
I20260926 16:43:22.489996 11813 leader_election.cc:288] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf [CANDIDATE]: Term 1 election: Requested vote from peers 98cdcd35007d45ebbbc6249314867b54 (127.9.238.194:33185), 29744506ffd04e47905a7022ab1cfdb6 (127.9.238.195:45795)
I20260926 16:43:22.490582 11600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aee2f8c06b94360a5e1efdcdeae342b" candidate_uuid: "07c4e314672c437a95c60566c0d262cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98cdcd35007d45ebbbc6249314867b54"
I20260926 16:43:22.490686 11736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aee2f8c06b94360a5e1efdcdeae342b" candidate_uuid: "07c4e314672c437a95c60566c0d262cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29744506ffd04e47905a7022ab1cfdb6"
I20260926 16:43:22.490968 11736 raft_consensus.cc:3037] T 9aee2f8c06b94360a5e1efdcdeae342b P 29744506ffd04e47905a7022ab1cfdb6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:22.490970 11600 raft_consensus.cc:3037] T 9aee2f8c06b94360a5e1efdcdeae342b P 98cdcd35007d45ebbbc6249314867b54 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:22.493580 11736 raft_consensus.cc:2454] T 9aee2f8c06b94360a5e1efdcdeae342b P 29744506ffd04e47905a7022ab1cfdb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07c4e314672c437a95c60566c0d262cf in term 1.
I20260926 16:43:22.494131 11387 leader_election.cc:302] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c4e314672c437a95c60566c0d262cf, 29744506ffd04e47905a7022ab1cfdb6; no voters:
I20260926 16:43:22.494436 11813 raft_consensus.cc:2781] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:22.494639 11813 raft_consensus.cc:686] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf [term 1 LEADER]: Becoming Leader. State: Replica: 07c4e314672c437a95c60566c0d262cf, State: Running, Role: LEADER
I20260926 16:43:22.494680 11600 raft_consensus.cc:2454] T 9aee2f8c06b94360a5e1efdcdeae342b P 98cdcd35007d45ebbbc6249314867b54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07c4e314672c437a95c60566c0d262cf in term 1.
I20260926 16:43:22.494896 11813 consensus_queue.cc:237] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } }
I20260926 16:43:22.497131 11836 tablet_replica.cc:439] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf: TxnStatusTablet state changed. Reason: New leader 07c4e314672c437a95c60566c0d262cf. Latest consensus state: current_term: 1 leader_uuid: "07c4e314672c437a95c60566c0d262cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } } }
I20260926 16:43:22.497678 11836 tablet_replica.cc:442] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:43:22.498520 11848 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 16:43:22.498791 11848 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 16:43:22.498791 11298 catalog_manager.cc:5495] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 07c4e314672c437a95c60566c0d262cf (127.9.238.193). New cstate: current_term: 1 leader_uuid: "07c4e314672c437a95c60566c0d262cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } health_report { overall_health: UNKNOWN } } }
I20260926 16:43:22.528782 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:22.519860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35614 (local address 127.9.238.193:42155)
0926 16:43:22.520119 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:22.520127 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:22.520162 (+ 35us) server_negotiation.cc:407] Connection header received
0926 16:43:22.520333 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:22.520342 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:22.520524 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:22.520756 (+ 232us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:43:22.521240 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.522677 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.524744 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.525533 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.525705 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:22.526358 (+ 653us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:43:22.527640 (+ 1282us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:43:22.527867 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:22.528270 (+ 403us) server_negotiation.cc:299] Negotiation successful
0926 16:43:22.528406 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 16:43:22.528831 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:22.519317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:42155 (local address 127.0.0.1:35614)
0926 16:43:22.519684 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:22.519709 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:22.519902 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:22.520804 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:22.520810 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:22.520825 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:43:22.521057 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:22.521069 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.522878 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.522884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:22.524552 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:22.524567 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.525740 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.525747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:22.525940 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:22.526037 (+ 97us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:43:22.527994 (+ 1957us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:43:22.528031 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 16:43:22.528154 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 16:43:22.528381 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 16:43:22.535527 11600 raft_consensus.cc:1261] T 9aee2f8c06b94360a5e1efdcdeae342b P 98cdcd35007d45ebbbc6249314867b54 [term 1 FOLLOWER]: Refusing update from remote peer 07c4e314672c437a95c60566c0d262cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:43:22.535640 11736 raft_consensus.cc:1261] T 9aee2f8c06b94360a5e1efdcdeae342b P 29744506ffd04e47905a7022ab1cfdb6 [term 1 FOLLOWER]: Refusing update from remote peer 07c4e314672c437a95c60566c0d262cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:43:22.537058 11813 consensus_queue.cc:1035] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260926 16:43:22.537906 11813 consensus_queue.cc:1035] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:43:22.550622 11815 tablet_replica.cc:439] T 9aee2f8c06b94360a5e1efdcdeae342b P 29744506ffd04e47905a7022ab1cfdb6: TxnStatusTablet state changed. Reason: New leader 07c4e314672c437a95c60566c0d262cf. Latest consensus state: current_term: 1 leader_uuid: "07c4e314672c437a95c60566c0d262cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } } }
I20260926 16:43:22.551033 11815 tablet_replica.cc:442] T 9aee2f8c06b94360a5e1efdcdeae342b P 29744506ffd04e47905a7022ab1cfdb6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:43:22.552023 11811 tablet_replica.cc:439] T 9aee2f8c06b94360a5e1efdcdeae342b P 98cdcd35007d45ebbbc6249314867b54: TxnStatusTablet state changed. Reason: New leader 07c4e314672c437a95c60566c0d262cf. Latest consensus state: current_term: 1 leader_uuid: "07c4e314672c437a95c60566c0d262cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } } }
I20260926 16:43:22.552315 11811 tablet_replica.cc:442] T 9aee2f8c06b94360a5e1efdcdeae342b P 98cdcd35007d45ebbbc6249314867b54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:43:22.556085 11813 tablet_replica.cc:439] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "07c4e314672c437a95c60566c0d262cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } } }
I20260926 16:43:22.556509 11813 tablet_replica.cc:442] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:43:22.556699 11820 mvcc.cc:205] Tried to move back new op lower bound from 7333646346375352320 to 7333646346222198784. Current Snapshot: MvccSnapshot[applied={T|T < 7333646346375352320 or (T in {7333646346375352320})}]
I20260926 16:43:22.560350 11813 tablet_replica.cc:439] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "07c4e314672c437a95c60566c0d262cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } } }
I20260926 16:43:22.560600 11813 tablet_replica.cc:442] T 9aee2f8c06b94360a5e1efdcdeae342b P 07c4e314672c437a95c60566c0d262cf: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:43:22.562009 11818 mvcc.cc:205] Tried to move back new op lower bound from 7333646346375352320 to 7333646346222198784. Current Snapshot: MvccSnapshot[applied={T|T < 7333646346375352320}]
I20260926 16:43:22.562628 11815 tablet_replica.cc:439] T 9aee2f8c06b94360a5e1efdcdeae342b P 29744506ffd04e47905a7022ab1cfdb6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "07c4e314672c437a95c60566c0d262cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } } }
I20260926 16:43:22.562922 11815 tablet_replica.cc:442] T 9aee2f8c06b94360a5e1efdcdeae342b P 29744506ffd04e47905a7022ab1cfdb6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:43:22.575730 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:22.566705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35628 (local address 127.9.238.193:42155)
0926 16:43:22.566877 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:22.566883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:22.566901 (+ 18us) server_negotiation.cc:407] Connection header received
0926 16:43:22.566988 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:22.566993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:22.567070 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:22.567163 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:43:22.567726 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.568956 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.571379 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.572289 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.572420 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:22.574434 (+ 2014us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:43:22.574582 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:43:22.574666 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:22.575272 (+ 606us) server_negotiation.cc:299] Negotiation successful
0926 16:43:22.575355 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 16:43:22.580065 11811 tablet_replica.cc:439] T 9aee2f8c06b94360a5e1efdcdeae342b P 98cdcd35007d45ebbbc6249314867b54: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "07c4e314672c437a95c60566c0d262cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07c4e314672c437a95c60566c0d262cf" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 42155 } } peers { permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 } } peers { permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 } } }
I20260926 16:43:22.580354 11811 tablet_replica.cc:442] T 9aee2f8c06b94360a5e1efdcdeae342b P 98cdcd35007d45ebbbc6249314867b54: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:43:22.623842 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:22.607032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:56857 (local address 127.9.238.193:42155)
0926 16:43:22.607267 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:22.607273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:22.616581 (+ 9308us) server_negotiation.cc:407] Connection header received
0926 16:43:22.616884 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:22.616890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:22.616998 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:22.617130 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:43:22.618079 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.619312 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.621556 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.622238 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.622323 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:22.623010 (+ 687us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:43:22.623179 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:43:22.623232 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:22.623408 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 16:43:22.623491 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 16:43:22.624737 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:22.606521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:42155 (local address 127.9.238.193:56857)
0926 16:43:22.616349 (+ 9828us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:22.616388 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:22.616690 (+ 302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:22.617302 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:22.617307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:22.617319 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:43:22.617912 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:22.617924 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.619471 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.619476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:22.621376 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:22.621397 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.622551 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.622559 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:22.622762 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:22.622843 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:43:22.623304 (+ 461us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:43:22.623313 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 16:43:22.624327 (+ 1014us) client_negotiation.cc:240] Negotiation successful
0926 16:43:22.624440 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9506,"thread_start_us":9233,"threads_started":1}
I20260926 16:43:22.640477 11600 raft_consensus.cc:1261] T 3d33ad0ad6e146b092473f0ba7999c33 P 98cdcd35007d45ebbbc6249314867b54 [term 1 FOLLOWER]: Refusing update from remote peer 07c4e314672c437a95c60566c0d262cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:43:22.641427 11813 consensus_queue.cc:1035] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "98cdcd35007d45ebbbc6249314867b54" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:43:22.651036 11736 raft_consensus.cc:1261] T 3d33ad0ad6e146b092473f0ba7999c33 P 29744506ffd04e47905a7022ab1cfdb6 [term 1 FOLLOWER]: Refusing update from remote peer 07c4e314672c437a95c60566c0d262cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:43:22.653896 11836 consensus_queue.cc:1035] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "29744506ffd04e47905a7022ab1cfdb6" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:43:22.757937 11298 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:43:22.758198 11298 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:43:22.762290 11298 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17ee74c6a2a24bdcabd99741ead3b6fc: Sending DeleteTablet for 3 replicas of tablet 3d33ad0ad6e146b092473f0ba7999c33
I20260926 16:43:22.763363 11576 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d33ad0ad6e146b092473f0ba7999c33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:22 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:54510
I20260926 16:43:22.763597 11716 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d33ad0ad6e146b092473f0ba7999c33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:22 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:46844
I20260926 16:43:22.764213 11863 tablet_replica.cc:330] T 3d33ad0ad6e146b092473f0ba7999c33 P 98cdcd35007d45ebbbc6249314867b54: stopping tablet replica
I20260926 16:43:22.764477 11863 raft_consensus.cc:2229] T 3d33ad0ad6e146b092473f0ba7999c33 P 98cdcd35007d45ebbbc6249314867b54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:43:22.764794 11863 raft_consensus.cc:2258] T 3d33ad0ad6e146b092473f0ba7999c33 P 98cdcd35007d45ebbbc6249314867b54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:43:22.766130 11863 ts_tablet_manager.cc:1905] T 3d33ad0ad6e146b092473f0ba7999c33 P 98cdcd35007d45ebbbc6249314867b54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:43:22.766405 11436 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d33ad0ad6e146b092473f0ba7999c33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:22 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:35608
I20260926 16:43:22.776961 11863 ts_tablet_manager.cc:1918] T 3d33ad0ad6e146b092473f0ba7999c33 P 98cdcd35007d45ebbbc6249314867b54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:43:22.777215 11863 log.cc:1196] T 3d33ad0ad6e146b092473f0ba7999c33 P 98cdcd35007d45ebbbc6249314867b54: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-1/wal/wals/3d33ad0ad6e146b092473f0ba7999c33
I20260926 16:43:22.777928 11863 ts_tablet_manager.cc:1939] T 3d33ad0ad6e146b092473f0ba7999c33 P 98cdcd35007d45ebbbc6249314867b54: Deleting consensus metadata
I20260926 16:43:22.779431 11286 catalog_manager.cc:4841] TS 98cdcd35007d45ebbbc6249314867b54 (127.9.238.194:33185): tablet 3d33ad0ad6e146b092473f0ba7999c33 (table test-table [id=f8fc35aec0db4729bd6105ba16a08399]) successfully deleted
I20260926 16:43:22.780926 11866 tablet_replica.cc:330] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf: stopping tablet replica
I20260926 16:43:22.781217 11866 raft_consensus.cc:2229] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:43:22.781761 11866 raft_consensus.cc:2258] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:43:22.783458 11864 tablet_replica.cc:330] T 3d33ad0ad6e146b092473f0ba7999c33 P 29744506ffd04e47905a7022ab1cfdb6: stopping tablet replica
I20260926 16:43:22.783771 11864 raft_consensus.cc:2229] T 3d33ad0ad6e146b092473f0ba7999c33 P 29744506ffd04e47905a7022ab1cfdb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:43:22.784086 11864 raft_consensus.cc:2258] T 3d33ad0ad6e146b092473f0ba7999c33 P 29744506ffd04e47905a7022ab1cfdb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:43:22.785274 11866 ts_tablet_manager.cc:1905] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:43:22.793457 11864 ts_tablet_manager.cc:1905] T 3d33ad0ad6e146b092473f0ba7999c33 P 29744506ffd04e47905a7022ab1cfdb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:43:22.810973 11864 ts_tablet_manager.cc:1918] T 3d33ad0ad6e146b092473f0ba7999c33 P 29744506ffd04e47905a7022ab1cfdb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:43:22.811206 11864 log.cc:1196] T 3d33ad0ad6e146b092473f0ba7999c33 P 29744506ffd04e47905a7022ab1cfdb6: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-2/wal/wals/3d33ad0ad6e146b092473f0ba7999c33
I20260926 16:43:22.811961 11864 ts_tablet_manager.cc:1939] T 3d33ad0ad6e146b092473f0ba7999c33 P 29744506ffd04e47905a7022ab1cfdb6: Deleting consensus metadata
I20260926 16:43:22.813618 11284 catalog_manager.cc:4841] TS 29744506ffd04e47905a7022ab1cfdb6 (127.9.238.195:45795): tablet 3d33ad0ad6e146b092473f0ba7999c33 (table test-table [id=f8fc35aec0db4729bd6105ba16a08399]) successfully deleted
I20260926 16:43:22.816598 11866 ts_tablet_manager.cc:1918] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:43:22.816782 11866 log.cc:1196] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790440988577176-10171-0/minicluster-data/ts-0/wal/wals/3d33ad0ad6e146b092473f0ba7999c33
I20260926 16:43:22.817430 11866 ts_tablet_manager.cc:1939] T 3d33ad0ad6e146b092473f0ba7999c33 P 07c4e314672c437a95c60566c0d262cf: Deleting consensus metadata
I20260926 16:43:22.819114 11286 catalog_manager.cc:4841] TS 07c4e314672c437a95c60566c0d262cf (127.9.238.193:42155): tablet 3d33ad0ad6e146b092473f0ba7999c33 (table test-table [id=f8fc35aec0db4729bd6105ba16a08399]) successfully deleted
Sep 26 16:43:22 dist-test-slave-wt2j krb5kdc[11254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441001, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.238.193@KRBTEST.COM
I20260926 16:43:22.838704 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:22.782952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35640 (local address 127.9.238.193:42155)
0926 16:43:22.783171 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:22.783179 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:22.814811 (+ 31632us) server_negotiation.cc:407] Connection header received
0926 16:43:22.815042 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:22.815048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:22.815157 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:22.815307 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:22.817726 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.819000 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.821037 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.821891 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.822083 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:22.827511 (+ 5428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:22.827572 (+ 61us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:22.827642 (+ 70us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:22.827697 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:22.832544 (+ 4847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:22.834286 (+ 1742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:22.834301 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:22.834306 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:22.834607 (+ 301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:22.835207 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:22.835212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:22.835215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:22.835901 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:22.836251 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:22.837971 (+ 1720us) server_negotiation.cc:299] Negotiation successful
0926 16:43:22.838288 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 16:43:22.840247 11501 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35640
I20260926 16:43:22.865684 11674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:22.845323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43066 (local address 127.9.238.254:33509)
0926 16:43:22.845545 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:22.845551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:22.848241 (+ 2690us) server_negotiation.cc:407] Connection header received
0926 16:43:22.848441 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:22.848448 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:22.848564 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:22.848706 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:22.850669 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.851864 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.853798 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.854411 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.854515 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:22.858312 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:22.858319 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:22.858324 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:22.858369 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:22.861963 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:22.862987 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:22.862994 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:22.862998 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:22.863170 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:22.863671 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:22.863679 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:22.863684 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:22.864077 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:22.864293 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:22.864952 (+ 659us) server_negotiation.cc:299] Negotiation successful
0926 16:43:22.865313 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260926 16:43:22.867532 11298 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43066
I20260926 16:43:22.870937 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 11370
I20260926 16:43:22.904670 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:22.872592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35646 (local address 127.9.238.193:42155)
0926 16:43:22.872810 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:22.872817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:22.886247 (+ 13430us) server_negotiation.cc:407] Connection header received
0926 16:43:22.886454 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:22.886460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:22.886578 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:22.886767 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:22.888518 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.890189 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.892194 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.893106 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.893352 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:22.896587 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:22.896595 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:22.896601 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:22.896661 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:22.901200 (+ 4539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:22.902182 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:22.902188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:22.902192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:22.902392 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:22.902848 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:22.902853 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:22.902857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:22.903260 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:22.903625 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:22.903968 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 16:43:22.904254 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260926 16:43:22.905377 11501 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35646
W20260926 16:43:22.906050 10171 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 11370: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:43:22.906308 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 11370
I20260926 16:43:22.924057 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 11510
Sep 26 16:43:22 dist-test-slave-wt2j krb5kdc[11254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441001, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.238.194@KRBTEST.COM
I20260926 16:43:22.943188 11871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:22.925107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54514 (local address 127.9.238.194:33185)
0926 16:43:22.925649 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:22.925654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:22.925672 (+ 18us) server_negotiation.cc:407] Connection header received
0926 16:43:22.925736 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:22.925739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:22.925812 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:22.925915 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:22.927390 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.928520 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.929854 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.930352 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.930503 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:22.935046 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:22.935067 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:22.935085 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:22.935128 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:22.938478 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:22.939830 (+ 1352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:22.939846 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:22.939852 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:22.940102 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:22.940843 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:22.940852 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:22.940858 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:22.941521 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:22.941843 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:22.942222 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 16:43:22.942581 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":331,"threads_started":1}
W20260926 16:43:22.944098 11641 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54514
W20260926 16:43:22.944971 10171 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 11510: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:43:22.945413 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 11510
I20260926 16:43:22.964283 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 11650
Sep 26 16:43:22 dist-test-slave-wt2j krb5kdc[11254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441001, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.238.195@KRBTEST.COM
I20260926 16:43:22.983070 11872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:22.965237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46866 (local address 127.9.238.195:45795)
0926 16:43:22.966141 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:22.966147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:22.966164 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:43:22.966229 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:22.966233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:22.966321 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:22.966442 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:22.967873 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.968658 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.969962 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:22.970454 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:22.970574 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:22.975691 (+ 5117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:22.975712 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:22.975735 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:22.975788 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:22.979384 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:22.980467 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:22.980477 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:22.980480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:22.980656 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:22.981105 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:22.981112 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:22.981115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:22.981602 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:22.981964 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:22.982321 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 16:43:22.982601 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":794,"thread_start_us":630,"threads_started":1}
W20260926 16:43:22.984148 11781 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46866
W20260926 16:43:22.985033 10171 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 11650: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:43:22.985257 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 11650
I20260926 16:43:23.007721 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 11267
I20260926 16:43:23.009341 11339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:43:23.128829 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 11267
2026-09-26T16:43:23Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5233 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:43:23 dist-test-slave-wt2j krb5kdc[11876](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:43:23 dist-test-slave-wt2j krb5kdc[11876](info): set up 2 sockets
Sep 26 16:43:23 dist-test-slave-wt2j krb5kdc[11876](info): commencing operation
krb5kdc: starting...
W20260926 16:43:24.289175 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:43:24 dist-test-slave-wt2j krb5kdc[11876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:43:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:43:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:43:24.515705 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:45877
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:32971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:45877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:24.873502 11889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:24.873890 11889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:24.874030 11889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:24.881655 11889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:43:24.881774 11889 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 16:43:24.881815 11889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:24.881870 11889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:24.881912 11889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:43:24.881978 11889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:43:24.892268 11889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:32971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:45877
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:45877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:24.893612 11889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:24.895023 11889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:24.908826 11897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:24.909034 11894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:24.908910 11895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:24.910386 11889 server_base.cc:1034] running on GCE node
I20260926 16:43:24.911525 11889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:24.913658 11889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:24.914965 11889 hybrid_clock.cc:648] HybridClock initialized: now 1790441004914892 us; error 42 us; skew 500 ppm
I20260926 16:43:24.915419 11889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:24 dist-test-slave-wt2j krb5kdc[11876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441004, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:24.929934 11889 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:43:24.932118 11889 webserver.cc:460] Webserver started at http://127.9.238.254:44041/ using document root <none> and password file <none>
I20260926 16:43:24.933087 11889 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:24.933233 11889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:24.933615 11889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:24.937498 11889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "77c9e7c806ed4ecd91a120bde5c165ef"
format_stamp: "Formatted at 2026-09-26 16:43:24 on dist-test-slave-wt2j"
server_key: "0eded3e1bea6637e3ffb67c3bf8c3743"
server_key_iv: "e38ae979f21b69eabefcb966db0cff2f"
server_key_version: "encryptionkey@0"
I20260926 16:43:24.938443 11889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "77c9e7c806ed4ecd91a120bde5c165ef"
format_stamp: "Formatted at 2026-09-26 16:43:24 on dist-test-slave-wt2j"
server_key: "0eded3e1bea6637e3ffb67c3bf8c3743"
server_key_iv: "e38ae979f21b69eabefcb966db0cff2f"
server_key_version: "encryptionkey@0"
I20260926 16:43:24.945410 11889 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 16:43:24.950845 11904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:24.953265 11889 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260926 16:43:24.953526 11889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "77c9e7c806ed4ecd91a120bde5c165ef"
format_stamp: "Formatted at 2026-09-26 16:43:24 on dist-test-slave-wt2j"
server_key: "0eded3e1bea6637e3ffb67c3bf8c3743"
server_key_iv: "e38ae979f21b69eabefcb966db0cff2f"
server_key_version: "encryptionkey@0"
I20260926 16:43:24.953806 11889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:24.980914 11889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:24.986209 11889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:24.986533 11889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:25.014038 11889 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:45877
I20260926 16:43:25.014104 11965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:45877 every 8 connection(s)
I20260926 16:43:25.015858 11889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:43:25.017393 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 11889
I20260926 16:43:25.017629 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:43:25.018357 10171 external_mini_cluster.cc:1428] Setting key 24f4f9cb948c495415d14de995a61d69
I20260926 16:43:25.021405 11966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:25.034337 11966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c9e7c806ed4ecd91a120bde5c165ef: Bootstrap starting.
Sep 26 16:43:25 dist-test-slave-wt2j krb5kdc[11876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:25.037910 11966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77c9e7c806ed4ecd91a120bde5c165ef: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:25.039198 11966 log.cc:824] T 00000000000000000000000000000000 P 77c9e7c806ed4ecd91a120bde5c165ef: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:25.043222 11966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c9e7c806ed4ecd91a120bde5c165ef: No bootstrap required, opened a new log
I20260926 16:43:25.047801 11969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:25.020913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54874 (local address 127.9.238.254:45877)
0926 16:43:25.022059 (+ 1146us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:25.022082 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:25.022595 (+ 513us) server_negotiation.cc:407] Connection header received
0926 16:43:25.024860 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:25.024886 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:25.025469 (+ 583us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:25.026016 (+ 547us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:25.027977 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:25.029545 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:25.030962 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:25.032038 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:25.032336 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:25.037669 (+ 5333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:25.037680 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:25.037702 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:25.037758 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:25.042045 (+ 4287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:25.043222 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:25.043231 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:25.043240 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:25.043661 (+ 421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:25.044185 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:25.044191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:25.044195 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:25.044955 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:25.045340 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:25.045853 (+ 513us) server_negotiation.cc:299] Negotiation successful
0926 16:43:25.046390 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":812,"thread_start_us":525,"threads_started":1}
I20260926 16:43:25.052409 11966 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77c9e7c806ed4ecd91a120bde5c165ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c9e7c806ed4ecd91a120bde5c165ef" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 45877 } }
I20260926 16:43:25.052812 11966 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77c9e7c806ed4ecd91a120bde5c165ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:25.052920 11966 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77c9e7c806ed4ecd91a120bde5c165ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c9e7c806ed4ecd91a120bde5c165ef, State: Initialized, Role: FOLLOWER
I20260926 16:43:25.053496 11966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77c9e7c806ed4ecd91a120bde5c165ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c9e7c806ed4ecd91a120bde5c165ef" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 45877 } }
I20260926 16:43:25.053757 11966 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77c9e7c806ed4ecd91a120bde5c165ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:43:25.053900 11966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77c9e7c806ed4ecd91a120bde5c165ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:43:25.054095 11966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77c9e7c806ed4ecd91a120bde5c165ef [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:25.056604 11966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77c9e7c806ed4ecd91a120bde5c165ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c9e7c806ed4ecd91a120bde5c165ef" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 45877 } }
I20260926 16:43:25.057134 11966 leader_election.cc:302] T 00000000000000000000000000000000 P 77c9e7c806ed4ecd91a120bde5c165ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77c9e7c806ed4ecd91a120bde5c165ef; no voters:
I20260926 16:43:25.057780 11966 leader_election.cc:288] T 00000000000000000000000000000000 P 77c9e7c806ed4ecd91a120bde5c165ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:43:25.058027 11971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77c9e7c806ed4ecd91a120bde5c165ef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:25.058969 11971 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77c9e7c806ed4ecd91a120bde5c165ef [term 1 LEADER]: Becoming Leader. State: Replica: 77c9e7c806ed4ecd91a120bde5c165ef, State: Running, Role: LEADER
I20260926 16:43:25.059769 11971 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77c9e7c806ed4ecd91a120bde5c165ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c9e7c806ed4ecd91a120bde5c165ef" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 45877 } }
I20260926 16:43:25.060364 11966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77c9e7c806ed4ecd91a120bde5c165ef [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:43:25.062888 11973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c9e7c806ed4ecd91a120bde5c165ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77c9e7c806ed4ecd91a120bde5c165ef. Latest consensus state: current_term: 1 leader_uuid: "77c9e7c806ed4ecd91a120bde5c165ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c9e7c806ed4ecd91a120bde5c165ef" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 45877 } } }
I20260926 16:43:25.063382 11973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c9e7c806ed4ecd91a120bde5c165ef [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:25.064203 11972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c9e7c806ed4ecd91a120bde5c165ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77c9e7c806ed4ecd91a120bde5c165ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c9e7c806ed4ecd91a120bde5c165ef" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 45877 } } }
I20260926 16:43:25.064570 11972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c9e7c806ed4ecd91a120bde5c165ef [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:25.067374 11978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:43:25.073625 11978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:43:25.084246 11978 catalog_manager.cc:1347] Generated new cluster ID: 8f23b3c2da044614a2f7c957da5d68f5
I20260926 16:43:25.084401 11978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:43:25.101158 11978 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:43:25.102867 11978 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:43:25.121239 11978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77c9e7c806ed4ecd91a120bde5c165ef: Generated new TSK 0
I20260926 16:43:25.122355 11978 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
I20260926 16:43:25.196727 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:0
--local_ip_for_outbound_sockets=127.9.238.193
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:45877
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:32971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:25.618532 11992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:25.619032 11992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:25.619253 11992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:25.629204 11992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:25.629417 11992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:25.629597 11992 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 16:43:25.629693 11992 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:43:25.629837 11992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:43:25.643392 11992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:32971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.238.254:45877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:25.645592 11992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:25.647646 11992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:25.665518 11997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:25.665886 11998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:25.666242 12000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:25.667105 11992 server_base.cc:1034] running on GCE node
I20260926 16:43:25.668087 11992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:25.669549 11992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:25.670851 11992 hybrid_clock.cc:648] HybridClock initialized: now 1790441005670760 us; error 78 us; skew 500 ppm
I20260926 16:43:25.671362 11992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:25 dist-test-slave-wt2j krb5kdc[11876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441005, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:25.676833 11992 init.cc:377] Logged in from keytab as kudu/127.9.238.193@KRBTEST.COM (short username kudu)
I20260926 16:43:25.679365 11992 webserver.cc:460] Webserver started at http://127.9.238.193:38521/ using document root <none> and password file <none>
I20260926 16:43:25.680480 11992 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:25.680621 11992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:25.680966 11992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:25.685042 11992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-0/data/instance:
uuid: "752d81f769294f1183734d69013940f2"
format_stamp: "Formatted at 2026-09-26 16:43:25 on dist-test-slave-wt2j"
server_key: "04cc738700f48c13df3f22a45cbff2dc"
server_key_iv: "3d1357a3dfff08e8c6896350286fa57d"
server_key_version: "encryptionkey@0"
I20260926 16:43:25.685998 11992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance:
uuid: "752d81f769294f1183734d69013940f2"
format_stamp: "Formatted at 2026-09-26 16:43:25 on dist-test-slave-wt2j"
server_key: "04cc738700f48c13df3f22a45cbff2dc"
server_key_iv: "3d1357a3dfff08e8c6896350286fa57d"
server_key_version: "encryptionkey@0"
I20260926 16:43:25.693341 11992 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260926 16:43:25.699290 12007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:25.701622 11992 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260926 16:43:25.701866 11992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "752d81f769294f1183734d69013940f2"
format_stamp: "Formatted at 2026-09-26 16:43:25 on dist-test-slave-wt2j"
server_key: "04cc738700f48c13df3f22a45cbff2dc"
server_key_iv: "3d1357a3dfff08e8c6896350286fa57d"
server_key_version: "encryptionkey@0"
I20260926 16:43:25.702093 11992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:25.733641 11992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:25.740367 11992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:25.740680 11992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:25.744185 11992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:25.744304 11992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:25.744454 11992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:25.744555 11992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:43:25 dist-test-slave-wt2j krb5kdc[11876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441005, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:25.774971 12016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:25.745173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:55829 (local address 127.9.238.254:45877)
0926 16:43:25.745691 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:25.745698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:25.747255 (+ 1557us) server_negotiation.cc:407] Connection header received
0926 16:43:25.750817 (+ 3562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:25.750824 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:25.750936 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:25.751107 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:25.754268 (+ 3161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:25.755346 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:25.759075 (+ 3729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:25.759841 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:25.760065 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:25.766411 (+ 6346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:25.766436 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:25.766446 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:25.766523 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:25.770278 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:25.771172 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:25.771177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:25.771180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:25.771311 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:25.772119 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:25.772130 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:25.772137 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:25.772793 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:25.773048 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:25.773949 (+ 901us) server_negotiation.cc:299] Negotiation successful
0926 16:43:25.774345 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":273,"threads_started":1}
I20260926 16:43:25.775539 12017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:25.745870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:45877 (local address 127.9.238.193:55829)
0926 16:43:25.746972 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:25.747040 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:25.750297 (+ 3257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:25.751701 (+ 1404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:25.751719 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:25.752778 (+ 1059us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:25.753827 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:25.753862 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:25.755677 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:25.755686 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:25.758821 (+ 3135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:25.758846 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:25.760303 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:25.760316 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:25.760655 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:25.761833 (+ 1178us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:25.761882 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:25.765974 (+ 4092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:25.770482 (+ 4508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:25.770501 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:25.770529 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:25.771015 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:25.771434 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:25.771439 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:25.771443 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:25.771859 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:25.773021 (+ 1162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:25.773030 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:25.773447 (+ 417us) client_negotiation.cc:769] Sending connection context
0926 16:43:25.773722 (+ 275us) client_negotiation.cc:240] Negotiation successful
0926 16:43:25.774159 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":400,"threads_started":1}
I20260926 16:43:25.802685 12070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:25.782750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:45877 (local address 127.9.238.193:51639)
0926 16:43:25.783461 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:25.783478 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:25.783793 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:25.784373 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:25.784379 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:25.785382 (+ 1003us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:25.786160 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:25.786185 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:25.787863 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:25.787875 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:25.790016 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:25.790046 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:25.791516 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:25.791525 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:25.791833 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:25.792749 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:25.792782 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:25.794258 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:25.798574 (+ 4316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:25.798581 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:25.798587 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:25.799337 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:25.799890 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:25.799897 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:25.799902 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:25.800130 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:25.801026 (+ 896us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:25.801037 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:25.801307 (+ 270us) client_negotiation.cc:769] Sending connection context
0926 16:43:25.801501 (+ 194us) client_negotiation.cc:240] Negotiation successful
0926 16:43:25.801969 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":370,"threads_started":1}
I20260926 16:43:25.803973 12016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:25.782892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:51639 (local address 127.9.238.254:45877)
0926 16:43:25.783199 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:25.783208 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:25.783626 (+ 418us) server_negotiation.cc:407] Connection header received
0926 16:43:25.784005 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:25.784015 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:25.784139 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:25.784313 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:25.786428 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:25.787675 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:25.790254 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:25.790994 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:25.791154 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:25.794547 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:25.794556 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:25.794562 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:25.794609 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:25.798371 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:25.799529 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:25.799536 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:25.799541 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:25.799735 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:25.800294 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:25.800310 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:25.800314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:25.800763 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:25.800930 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:25.803216 (+ 2286us) server_negotiation.cc:299] Negotiation successful
0926 16:43:25.803503 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260926 16:43:25.806994 11992 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:34407
I20260926 16:43:25.807017 12127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:34407 every 8 connection(s)
I20260926 16:43:25.809069 11992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:43:25.811915 12128 heartbeater.cc:347] Connected to a master server at 127.9.238.254:45877
I20260926 16:43:25.812292 12128 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:25.813076 12128 heartbeater.cc:511] Master 127.9.238.254:45877 requested a full tablet report, sending...
I20260926 16:43:25.815829 11921 ts_manager.cc:194] Registered new tserver with Master: 752d81f769294f1183734d69013940f2 (127.9.238.193:34407)
I20260926 16:43:25.817430 11921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.193@KRBTEST.COM'} at 127.9.238.193:55829
I20260926 16:43:25.818467 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 11992
I20260926 16:43:25.818735 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance
I20260926 16:43:25.819679 10171 external_mini_cluster.cc:1428] Setting key 2ee659ad2adea639f515088e7695d8f6
WARNING: no policy specified for kudu/127.9.238.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.194@KRBTEST.COM" created.
I20260926 16:43:26.821897 12128 heartbeater.cc:503] Master 127.9.238.254:45877 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
W20260926 16:43:27.248695 10171 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.9.238.194: real 1.426s user 0.003s sys 0.011s
I20260926 16:43:27.249653 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:0
--local_ip_for_outbound_sockets=127.9.238.194
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:45877
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:32971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:27.663946 12132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:27.664316 12132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:27.664556 12132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:27.672956 12132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:27.673120 12132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:27.673285 12132 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 16:43:27.673354 12132 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:43:27.673470 12132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:43:27.687201 12132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:32971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.238.254:45877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:27.689148 12132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:27.690922 12132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:27.708523 12138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:27.708616 12140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:27.708648 12137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:27.710165 12132 server_base.cc:1034] running on GCE node
I20260926 16:43:27.711045 12132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:27.712476 12132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:27.713752 12132 hybrid_clock.cc:648] HybridClock initialized: now 1790441007713697 us; error 41 us; skew 500 ppm
I20260926 16:43:27.714200 12132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:27 dist-test-slave-wt2j krb5kdc[11876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441007, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:27.719110 12132 init.cc:377] Logged in from keytab as kudu/127.9.238.194@KRBTEST.COM (short username kudu)
I20260926 16:43:27.723133 12132 webserver.cc:460] Webserver started at http://127.9.238.194:44459/ using document root <none> and password file <none>
I20260926 16:43:27.724121 12132 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:27.724288 12132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:27.724684 12132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:27.728413 12132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-1/data/instance:
uuid: "d0139f9f3e4e4235ac89980b174840fe"
format_stamp: "Formatted at 2026-09-26 16:43:27 on dist-test-slave-wt2j"
server_key: "2b3b5d05a0e70696bb3eceab4ef322e2"
server_key_iv: "7a07e10ce98337694aac78efab36ba83"
server_key_version: "encryptionkey@0"
I20260926 16:43:27.729346 12132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance:
uuid: "d0139f9f3e4e4235ac89980b174840fe"
format_stamp: "Formatted at 2026-09-26 16:43:27 on dist-test-slave-wt2j"
server_key: "2b3b5d05a0e70696bb3eceab4ef322e2"
server_key_iv: "7a07e10ce98337694aac78efab36ba83"
server_key_version: "encryptionkey@0"
I20260926 16:43:27.736523 12132 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260926 16:43:27.741168 12147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:27.743273 12132 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:43:27.743532 12132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "d0139f9f3e4e4235ac89980b174840fe"
format_stamp: "Formatted at 2026-09-26 16:43:27 on dist-test-slave-wt2j"
server_key: "2b3b5d05a0e70696bb3eceab4ef322e2"
server_key_iv: "7a07e10ce98337694aac78efab36ba83"
server_key_version: "encryptionkey@0"
I20260926 16:43:27.743847 12132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:27.771056 12132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:27.776647 12132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:27.777038 12132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:27.780884 12132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:27.781047 12132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:27.781220 12132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:27.781332 12132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:43:27 dist-test-slave-wt2j krb5kdc[11876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441007, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:27.815487 12156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:27.782346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:50947 (local address 127.9.238.254:45877)
0926 16:43:27.783014 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:27.783022 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:27.784746 (+ 1724us) server_negotiation.cc:407] Connection header received
0926 16:43:27.788376 (+ 3630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:27.788388 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:27.788531 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:27.788751 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:27.791776 (+ 3025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:27.793092 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:27.795246 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:27.796198 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:27.796426 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:27.804695 (+ 8269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:27.804705 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:27.804713 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:27.804779 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:27.810066 (+ 5287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:27.811622 (+ 1556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:27.811630 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:27.811639 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:27.811845 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:27.812785 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:27.812793 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:27.812799 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:27.813250 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:27.813445 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:27.814831 (+ 1386us) server_negotiation.cc:299] Negotiation successful
0926 16:43:27.815089 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":324,"threads_started":1}
I20260926 16:43:27.818018 12157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:27.783265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:45877 (local address 127.9.238.194:50947)
0926 16:43:27.784475 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:27.784537 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:27.787839 (+ 3302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:27.789026 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:27.789036 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:27.790138 (+ 1102us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:27.791441 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:27.791480 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:27.793305 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:27.793316 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:27.794984 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:27.795001 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:27.796961 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:27.796973 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:27.797525 (+ 552us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:27.798722 (+ 1197us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:27.798774 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:27.804253 (+ 5479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:27.810306 (+ 6053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:27.810324 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:27.810354 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:27.811331 (+ 977us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:27.812050 (+ 719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:27.812060 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:27.812063 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:27.812560 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:27.813423 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:27.813434 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:27.814199 (+ 765us) client_negotiation.cc:769] Sending connection context
0926 16:43:27.815787 (+ 1588us) client_negotiation.cc:240] Negotiation successful
0926 16:43:27.816512 (+ 725us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"thread_start_us":536,"threads_started":1}
I20260926 16:43:27.856750 12156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:27.833488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:40587 (local address 127.9.238.254:45877)
0926 16:43:27.833791 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:27.833801 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:27.834888 (+ 1087us) server_negotiation.cc:407] Connection header received
0926 16:43:27.835203 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:27.835208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:27.835303 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:27.835456 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:27.838182 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:27.839062 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:27.842850 (+ 3788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:27.843409 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:27.843539 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:27.847348 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:27.847353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:27.847357 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:27.847397 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:27.851684 (+ 4287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:27.853240 (+ 1556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:27.853249 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:27.853255 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:27.853449 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:27.854222 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:27.854234 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:27.854242 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:27.854992 (+ 750us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:27.855224 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:27.855808 (+ 584us) server_negotiation.cc:299] Negotiation successful
0926 16:43:27.856195 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260926 16:43:27.856976 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:27.833490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:45877 (local address 127.9.238.194:40587)
0926 16:43:27.834621 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:27.834655 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:27.834997 (+ 342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:27.835781 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:27.835791 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:27.837000 (+ 1209us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:27.837936 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:27.837956 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:27.839258 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:27.839268 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:27.842514 (+ 3246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:27.842541 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:27.843756 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:27.843768 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:27.844070 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:27.845138 (+ 1068us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:27.845173 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:27.847090 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:27.851983 (+ 4893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:27.851991 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:27.852001 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:27.852993 (+ 992us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:27.853687 (+ 694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:27.853693 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:27.853696 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:27.853993 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:27.855176 (+ 1183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:27.855184 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:27.855397 (+ 213us) client_negotiation.cc:769] Sending connection context
0926 16:43:27.855529 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 16:43:27.856261 (+ 732us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"thread_start_us":556,"threads_started":1}
I20260926 16:43:27.859730 12132 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:36687
I20260926 16:43:27.859748 12267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:36687 every 8 connection(s)
I20260926 16:43:27.861641 12132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:43:27.864723 12268 heartbeater.cc:347] Connected to a master server at 127.9.238.254:45877
I20260926 16:43:27.865042 12268 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:27.865924 12268 heartbeater.cc:511] Master 127.9.238.254:45877 requested a full tablet report, sending...
I20260926 16:43:27.866883 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 12132
I20260926 16:43:27.867158 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance
I20260926 16:43:27.867743 10171 external_mini_cluster.cc:1428] Setting key 0111772f8acd2cbc9114e48164d908c8
I20260926 16:43:27.867753 11921 ts_manager.cc:194] Registered new tserver with Master: d0139f9f3e4e4235ac89980b174840fe (127.9.238.194:36687)
I20260926 16:43:27.868752 11921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.194@KRBTEST.COM'} at 127.9.238.194:50947
WARNING: no policy specified for kudu/127.9.238.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.195@KRBTEST.COM" created.
I20260926 16:43:28.872742 12268 heartbeater.cc:503] Master 127.9.238.254:45877 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
W20260926 16:43:29.194728 10171 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.9.238.195: real 1.324s user 0.000s sys 0.010s
I20260926 16:43:29.195540 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:0
--local_ip_for_outbound_sockets=127.9.238.195
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:45877
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:32971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:29.576256 12272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:29.576570 12272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:29.576685 12272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:29.584637 12272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:29.584812 12272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:29.584908 12272 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 16:43:29.584955 12272 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:43:29.585033 12272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:43:29.595530 12272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:32971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.238.254:45877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:29.597122 12272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:29.598526 12272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:29.616385 12280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:29.616408 12277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:29.616432 12278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:29.617596 12272 server_base.cc:1034] running on GCE node
I20260926 16:43:29.618400 12272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:29.619777 12272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:29.621161 12272 hybrid_clock.cc:648] HybridClock initialized: now 1790441009621041 us; error 118 us; skew 500 ppm
I20260926 16:43:29.621606 12272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:29 dist-test-slave-wt2j krb5kdc[11876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441009, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:29.625733 12272 init.cc:377] Logged in from keytab as kudu/127.9.238.195@KRBTEST.COM (short username kudu)
I20260926 16:43:29.627736 12272 webserver.cc:460] Webserver started at http://127.9.238.195:37217/ using document root <none> and password file <none>
I20260926 16:43:29.628528 12272 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:29.628687 12272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:29.629035 12272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:29.632221 12272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-2/data/instance:
uuid: "7ce8ee95d02a4b278bb5a80c9e072be7"
format_stamp: "Formatted at 2026-09-26 16:43:29 on dist-test-slave-wt2j"
server_key: "e20c1186510f26c62d4bec56269af240"
server_key_iv: "f19f6cb13c43684bd62a3d712b01896f"
server_key_version: "encryptionkey@0"
I20260926 16:43:29.633033 12272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance:
uuid: "7ce8ee95d02a4b278bb5a80c9e072be7"
format_stamp: "Formatted at 2026-09-26 16:43:29 on dist-test-slave-wt2j"
server_key: "e20c1186510f26c62d4bec56269af240"
server_key_iv: "f19f6cb13c43684bd62a3d712b01896f"
server_key_version: "encryptionkey@0"
I20260926 16:43:29.639789 12272 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 16:43:29.644667 12287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:29.646598 12272 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:43:29.646855 12272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "7ce8ee95d02a4b278bb5a80c9e072be7"
format_stamp: "Formatted at 2026-09-26 16:43:29 on dist-test-slave-wt2j"
server_key: "e20c1186510f26c62d4bec56269af240"
server_key_iv: "f19f6cb13c43684bd62a3d712b01896f"
server_key_version: "encryptionkey@0"
I20260926 16:43:29.647106 12272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:29.665975 12272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:29.671423 12272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:29.671784 12272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:29.675352 12272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:29.675460 12272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:29.675623 12272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:29.675736 12272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 26 16:43:29 dist-test-slave-wt2j krb5kdc[11876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441009, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:29.712433 12296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:29.676583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:35039 (local address 127.9.238.254:45877)
0926 16:43:29.677233 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:29.677241 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:29.679105 (+ 1864us) server_negotiation.cc:407] Connection header received
0926 16:43:29.682476 (+ 3371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:29.682483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:29.682624 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:29.682786 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:29.686532 (+ 3746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:29.687996 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:29.692576 (+ 4580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:29.693481 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:29.693698 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:29.701973 (+ 8275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:29.701986 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:29.701995 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:29.702068 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:29.706717 (+ 4649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:29.708475 (+ 1758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:29.708485 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:29.708491 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:29.708750 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:29.709473 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:29.709480 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:29.709485 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:29.709949 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:29.710191 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:29.711646 (+ 1455us) server_negotiation.cc:299] Negotiation successful
0926 16:43:29.711967 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":327,"threads_started":1}
I20260926 16:43:29.714504 12297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:29.677525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:45877 (local address 127.9.238.195:35039)
0926 16:43:29.678797 (+ 1272us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:29.678881 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:29.681851 (+ 2970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:29.683255 (+ 1404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:29.683276 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:29.684574 (+ 1298us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:29.686104 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:29.686153 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:29.688319 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:29.688330 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:29.692172 (+ 3842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:29.692217 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:29.693921 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:29.693936 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:29.694625 (+ 689us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:29.696098 (+ 1473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:29.696163 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:29.701392 (+ 5229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:29.707091 (+ 5699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:29.707113 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:29.707151 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:29.708216 (+ 1065us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:29.708930 (+ 714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:29.708938 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:29.708945 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:29.709246 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:29.710151 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:29.710164 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:29.711006 (+ 842us) client_negotiation.cc:769] Sending connection context
0926 16:43:29.711466 (+ 460us) client_negotiation.cc:240] Negotiation successful
0926 16:43:29.712282 (+ 816us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"thread_start_us":557,"threads_started":1}
I20260926 16:43:29.735189 12272 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:44787
I20260926 16:43:29.735950 12407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:44787 every 8 connection(s)
I20260926 16:43:29.737231 12272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:43:29.740960 12408 heartbeater.cc:347] Connected to a master server at 127.9.238.254:45877
I20260926 16:43:29.741366 12408 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:29.742475 12408 heartbeater.cc:511] Master 127.9.238.254:45877 requested a full tablet report, sending...
I20260926 16:43:29.743172 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 12272
I20260926 16:43:29.743386 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance
I20260926 16:43:29.744047 10171 external_mini_cluster.cc:1428] Setting key c8263bac7b250cec0761c67c0cb0d86a
I20260926 16:43:29.745044 11920 ts_manager.cc:194] Registered new tserver with Master: 7ce8ee95d02a4b278bb5a80c9e072be7 (127.9.238.195:44787)
I20260926 16:43:29.746409 11920 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.195@KRBTEST.COM'} at 127.9.238.195:35039
I20260926 16:43:29.747913 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:43:29.749814 12397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:29.729762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:45877 (local address 127.9.238.195:38947)
0926 16:43:29.731025 (+ 1263us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:29.731053 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:29.731321 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:29.732260 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:29.732275 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:29.733260 (+ 985us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:29.733674 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:29.733689 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:29.735505 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:29.735510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:29.737572 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:29.737585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:29.738872 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:29.738882 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:29.739186 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:29.740105 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:29.740124 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:29.741186 (+ 1062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:29.745333 (+ 4147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:29.745338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:29.745341 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:29.745879 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:29.747358 (+ 1479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:29.747365 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:29.747370 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:29.747655 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:29.748342 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:29.748347 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:29.748624 (+ 277us) client_negotiation.cc:769] Sending connection context
0926 16:43:29.748776 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 16:43:29.749225 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1079,"thread_start_us":889,"threads_started":1}
I20260926 16:43:29.751418 12296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:29.729909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:38947 (local address 127.9.238.254:45877)
0926 16:43:29.730224 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:29.730235 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:29.731179 (+ 944us) server_negotiation.cc:407] Connection header received
0926 16:43:29.731664 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:29.731673 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:29.731806 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:29.732001 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:29.733946 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:29.735312 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:29.737814 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:29.738583 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:29.738858 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:29.741416 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:29.741422 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:29.741426 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:29.741490 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:29.745155 (+ 3665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:29.746045 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:29.746050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:29.746054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:29.746183 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:29.747808 (+ 1625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:29.747813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:29.747816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:29.748191 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:29.748355 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:29.750626 (+ 2271us) server_negotiation.cc:299] Negotiation successful
0926 16:43:29.750926 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
Sep 26 16:43:29 dist-test-slave-wt2j krb5kdc[11876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 16:43:29 dist-test-slave-wt2j krb5kdc[11876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:29.826721 12296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:29.808847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54878 (local address 127.9.238.254:45877)
0926 16:43:29.809156 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:29.809163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:29.809699 (+ 536us) server_negotiation.cc:407] Connection header received
0926 16:43:29.809903 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:29.809908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:29.809986 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:29.810116 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:29.812096 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:29.812853 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:29.814416 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:29.814841 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:29.814948 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:29.820003 (+ 5055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:29.820009 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:29.820013 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:29.820055 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:29.823305 (+ 3250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:29.824392 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:29.824396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:29.824398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:29.824552 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:29.825050 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:29.825054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:29.825056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:29.825357 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:29.825571 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:29.826082 (+ 511us) server_negotiation.cc:299] Negotiation successful
0926 16:43:29.826277 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260926 16:43:29.833005 11921 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:43:29.840238 11921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:43:29.874691 12424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:29.857487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39210 (local address 127.9.238.194:36687)
0926 16:43:29.860768 (+ 3281us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:29.860780 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:29.861728 (+ 948us) server_negotiation.cc:407] Connection header received
0926 16:43:29.862140 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:29.862151 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:29.862510 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:29.862764 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:29.864208 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:29.865806 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:29.870008 (+ 4202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:29.872660 (+ 2652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:29.872815 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:29.873156 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:29.874063 (+ 907us) server_negotiation.cc:299] Negotiation successful
0926 16:43:29.874320 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1569,"thread_start_us":3326,"threads_started":1}
I20260926 16:43:29.878321 12426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:29.859418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47518 (local address 127.9.238.193:34407)
0926 16:43:29.869272 (+ 9854us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:29.869282 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:29.869324 (+ 42us) server_negotiation.cc:407] Connection header received
0926 16:43:29.869455 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:29.869467 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:29.869814 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:29.870058 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:29.870902 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:29.872458 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:29.875642 (+ 3184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:29.877028 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:29.877192 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:29.877428 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:29.877771 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 16:43:29.877965 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9646,"thread_start_us":9465,"threads_started":1}
I20260926 16:43:29.878523 12425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:29.857225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:36687 (local address 127.0.0.1:39210)
0926 16:43:29.861600 (+ 4375us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:29.861622 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:29.861937 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:29.863105 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:29.863113 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:29.863456 (+ 343us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:29.864014 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:29.864027 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:29.866032 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:29.866036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:29.868104 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:29.868123 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:29.873072 (+ 4949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:29.873082 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:29.873541 (+ 459us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:29.873621 (+ 80us) client_negotiation.cc:769] Sending connection context
0926 16:43:29.873815 (+ 194us) client_negotiation.cc:240] Negotiation successful
0926 16:43:29.877937 (+ 4122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4215,"mutex_wait_us":735,"thread_start_us":11195,"threads_started":1}
I20260926 16:43:29.878643 12423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:29.858731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:34407 (local address 127.0.0.1:47518)
0926 16:43:29.868809 (+ 10078us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:29.868833 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:29.868959 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:29.870067 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:29.870073 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:29.870111 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:29.870616 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:29.870634 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:29.872687 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:29.872694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:29.875381 (+ 2687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:29.875396 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:29.877306 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:29.877314 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:29.877536 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:29.877575 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 16:43:29.877681 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 16:43:29.877927 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9896,"mutex_wait_us":4,"thread_start_us":2583,"threads_started":1}
I20260926 16:43:29.880057 12421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:29.856381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:44787 (local address 127.0.0.1:37022)
0926 16:43:29.858119 (+ 1738us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:29.858198 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:29.858482 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:29.870503 (+ 12021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:29.870510 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:29.870542 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:29.870918 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:29.870927 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:29.872526 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:29.872533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:29.875427 (+ 2894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:29.875443 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:29.878620 (+ 3177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:29.878636 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:29.878891 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:29.878982 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 16:43:29.879427 (+ 445us) client_negotiation.cc:240] Negotiation successful
0926 16:43:29.879624 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1248,"mutex_wait_us":103,"thread_start_us":1150,"threads_started":1}
I20260926 16:43:29.880115 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:29.856543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37022 (local address 127.9.238.195:44787)
0926 16:43:29.869706 (+ 13163us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:29.869715 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:29.869747 (+ 32us) server_negotiation.cc:407] Connection header received
0926 16:43:29.869897 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:29.869908 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:29.870238 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:29.870485 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:29.871160 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:29.872370 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:29.875710 (+ 3340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:29.877326 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:29.877481 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:29.877777 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:29.879514 (+ 1737us) server_negotiation.cc:299] Negotiation successful
0926 16:43:29.879771 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12970,"thread_start_us":12719,"threads_started":1}
I20260926 16:43:29.884855 12058 tablet_service.cc:1467] Processing CreateTablet for tablet 020d41b160b642d392e27503fb06c084 (DEFAULT_TABLE table=test-table [id=e793b92b7887404c86c8a34b125154ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:29.885094 12198 tablet_service.cc:1467] Processing CreateTablet for tablet 020d41b160b642d392e27503fb06c084 (DEFAULT_TABLE table=test-table [id=e793b92b7887404c86c8a34b125154ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:29.886765 12058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 020d41b160b642d392e27503fb06c084. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:29.886940 12198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 020d41b160b642d392e27503fb06c084. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:29.886700 12338 tablet_service.cc:1467] Processing CreateTablet for tablet 020d41b160b642d392e27503fb06c084 (DEFAULT_TABLE table=test-table [id=e793b92b7887404c86c8a34b125154ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:29.888394 12338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 020d41b160b642d392e27503fb06c084. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:29.901825 12427 tablet_bootstrap.cc:492] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe: Bootstrap starting.
I20260926 16:43:29.905592 12429 tablet_bootstrap.cc:492] T 020d41b160b642d392e27503fb06c084 P 7ce8ee95d02a4b278bb5a80c9e072be7: Bootstrap starting.
I20260926 16:43:29.906955 12427 tablet_bootstrap.cc:654] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:29.909001 12427 log.cc:824] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:29.910511 12429 tablet_bootstrap.cc:654] T 020d41b160b642d392e27503fb06c084 P 7ce8ee95d02a4b278bb5a80c9e072be7: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:29.912120 12429 log.cc:824] T 020d41b160b642d392e27503fb06c084 P 7ce8ee95d02a4b278bb5a80c9e072be7: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:29.912515 12428 tablet_bootstrap.cc:492] T 020d41b160b642d392e27503fb06c084 P 752d81f769294f1183734d69013940f2: Bootstrap starting.
I20260926 16:43:29.914959 12429 tablet_bootstrap.cc:492] T 020d41b160b642d392e27503fb06c084 P 7ce8ee95d02a4b278bb5a80c9e072be7: No bootstrap required, opened a new log
I20260926 16:43:29.915261 12429 ts_tablet_manager.cc:1397] T 020d41b160b642d392e27503fb06c084 P 7ce8ee95d02a4b278bb5a80c9e072be7: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 16:43:29.916599 12428 tablet_bootstrap.cc:654] T 020d41b160b642d392e27503fb06c084 P 752d81f769294f1183734d69013940f2: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:29.918219 12428 log.cc:824] T 020d41b160b642d392e27503fb06c084 P 752d81f769294f1183734d69013940f2: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:29.921905 12427 tablet_bootstrap.cc:492] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe: No bootstrap required, opened a new log
I20260926 16:43:29.922309 12427 ts_tablet_manager.cc:1397] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.005s
I20260926 16:43:29.923640 12429 raft_consensus.cc:348] T 020d41b160b642d392e27503fb06c084 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } } peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } }
I20260926 16:43:29.924176 12429 raft_consensus.cc:374] T 020d41b160b642d392e27503fb06c084 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:29.924295 12429 raft_consensus.cc:729] T 020d41b160b642d392e27503fb06c084 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ce8ee95d02a4b278bb5a80c9e072be7, State: Initialized, Role: FOLLOWER
I20260926 16:43:29.924611 12428 tablet_bootstrap.cc:492] T 020d41b160b642d392e27503fb06c084 P 752d81f769294f1183734d69013940f2: No bootstrap required, opened a new log
I20260926 16:43:29.924957 12428 ts_tablet_manager.cc:1397] T 020d41b160b642d392e27503fb06c084 P 752d81f769294f1183734d69013940f2: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260926 16:43:29.925101 12429 consensus_queue.cc:260] T 020d41b160b642d392e27503fb06c084 P 7ce8ee95d02a4b278bb5a80c9e072be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } } peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } }
I20260926 16:43:29.928575 12408 heartbeater.cc:503] Master 127.9.238.254:45877 was elected leader, sending a full tablet report...
I20260926 16:43:29.929209 12429 ts_tablet_manager.cc:1428] T 020d41b160b642d392e27503fb06c084 P 7ce8ee95d02a4b278bb5a80c9e072be7: Time spent starting tablet: real 0.014s user 0.008s sys 0.006s
I20260926 16:43:29.932775 12427 raft_consensus.cc:348] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } } peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } }
I20260926 16:43:29.933425 12427 raft_consensus.cc:374] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:29.933548 12427 raft_consensus.cc:729] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0139f9f3e4e4235ac89980b174840fe, State: Initialized, Role: FOLLOWER
I20260926 16:43:29.934173 12428 raft_consensus.cc:348] T 020d41b160b642d392e27503fb06c084 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } } peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } }
I20260926 16:43:29.934372 12427 consensus_queue.cc:260] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } } peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } }
I20260926 16:43:29.934609 12428 raft_consensus.cc:374] T 020d41b160b642d392e27503fb06c084 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:29.934708 12428 raft_consensus.cc:729] T 020d41b160b642d392e27503fb06c084 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 752d81f769294f1183734d69013940f2, State: Initialized, Role: FOLLOWER
I20260926 16:43:29.935525 12428 consensus_queue.cc:260] T 020d41b160b642d392e27503fb06c084 P 752d81f769294f1183734d69013940f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } } peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } }
I20260926 16:43:29.937525 12427 ts_tablet_manager.cc:1428] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260926 16:43:29.938876 12428 ts_tablet_manager.cc:1428] T 020d41b160b642d392e27503fb06c084 P 752d81f769294f1183734d69013940f2: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
W20260926 16:43:29.990731 12409 tablet.cc:2369] T 020d41b160b642d392e27503fb06c084 P 7ce8ee95d02a4b278bb5a80c9e072be7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:43:30.065238 12129 tablet.cc:2369] T 020d41b160b642d392e27503fb06c084 P 752d81f769294f1183734d69013940f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:43:30.084537 12434 raft_consensus.cc:482] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:43:30.084980 12434 raft_consensus.cc:504] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } } peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } }
I20260926 16:43:30.087328 12434 leader_election.cc:288] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ce8ee95d02a4b278bb5a80c9e072be7 (127.9.238.195:44787), 752d81f769294f1183734d69013940f2 (127.9.238.193:34407)
I20260926 16:43:30.097016 12426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:30.089269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:58633 (local address 127.9.238.193:34407)
0926 16:43:30.089582 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:30.089589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:30.089925 (+ 336us) server_negotiation.cc:407] Connection header received
0926 16:43:30.090114 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:30.090120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:30.090223 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:30.090385 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:30.091181 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.092141 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.094992 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.095915 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.096135 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:30.096219 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:30.096600 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 16:43:30.096744 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260926 16:43:30.097203 12437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:30.088969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:34407 (local address 127.9.238.194:58633)
0926 16:43:30.089781 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:30.089802 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:30.089955 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:30.090435 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:30.090439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:30.090492 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:30.090921 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:30.090931 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.092289 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.092297 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:30.094775 (+ 2478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:30.094794 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.096155 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.096162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:30.096396 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:30.096430 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 16:43:30.096559 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 16:43:30.096733 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":220,"thread_start_us":394,"threads_started":1}
I20260926 16:43:30.097692 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:30.087970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:34047 (local address 127.9.238.195:44787)
0926 16:43:30.088256 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:30.088265 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:30.089174 (+ 909us) server_negotiation.cc:407] Connection header received
0926 16:43:30.089562 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:30.089571 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:30.089683 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:30.089749 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:30.090889 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.091975 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.095082 (+ 3107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.096095 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.096497 (+ 402us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:30.096609 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:30.096960 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 16:43:30.097119 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 16:43:30.097913 12436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:30.087914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:44787 (local address 127.9.238.194:34047)
0926 16:43:30.088874 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:30.088898 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:30.089281 (+ 383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:30.090071 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:30.090077 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:30.090106 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:30.090701 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:30.090718 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.092135 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.092142 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:30.094832 (+ 2690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:30.094856 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.096503 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.096511 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:30.096712 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:30.096741 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 16:43:30.096922 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 16:43:30.097112 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":294,"thread_start_us":534,"threads_started":1}
I20260926 16:43:30.099340 12083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "020d41b160b642d392e27503fb06c084" candidate_uuid: "d0139f9f3e4e4235ac89980b174840fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "752d81f769294f1183734d69013940f2" is_pre_election: true
I20260926 16:43:30.099377 12358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "020d41b160b642d392e27503fb06c084" candidate_uuid: "d0139f9f3e4e4235ac89980b174840fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" is_pre_election: true
I20260926 16:43:30.100069 12358 raft_consensus.cc:2454] T 020d41b160b642d392e27503fb06c084 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0139f9f3e4e4235ac89980b174840fe in term 0.
I20260926 16:43:30.100124 12083 raft_consensus.cc:2454] T 020d41b160b642d392e27503fb06c084 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0139f9f3e4e4235ac89980b174840fe in term 0.
I20260926 16:43:30.100927 12149 leader_election.cc:302] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ce8ee95d02a4b278bb5a80c9e072be7, d0139f9f3e4e4235ac89980b174840fe; no voters:
I20260926 16:43:30.101402 12434 raft_consensus.cc:2781] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:43:30.101572 12434 raft_consensus.cc:482] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:43:30.101693 12434 raft_consensus.cc:3037] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:30.114601 12434 raft_consensus.cc:504] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } } peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } }
I20260926 16:43:30.115646 12434 leader_election.cc:288] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe [CANDIDATE]: Term 1 election: Requested vote from peers 7ce8ee95d02a4b278bb5a80c9e072be7 (127.9.238.195:44787), 752d81f769294f1183734d69013940f2 (127.9.238.193:34407)
W20260926 16:43:30.116317 12269 tablet.cc:2369] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:43:30.116577 12358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "020d41b160b642d392e27503fb06c084" candidate_uuid: "d0139f9f3e4e4235ac89980b174840fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7"
I20260926 16:43:30.116809 12358 raft_consensus.cc:3037] T 020d41b160b642d392e27503fb06c084 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:30.117332 12083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "020d41b160b642d392e27503fb06c084" candidate_uuid: "d0139f9f3e4e4235ac89980b174840fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "752d81f769294f1183734d69013940f2"
I20260926 16:43:30.117697 12083 raft_consensus.cc:3037] T 020d41b160b642d392e27503fb06c084 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:30.119604 12358 raft_consensus.cc:2454] T 020d41b160b642d392e27503fb06c084 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0139f9f3e4e4235ac89980b174840fe in term 1.
I20260926 16:43:30.120190 12149 leader_election.cc:302] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ce8ee95d02a4b278bb5a80c9e072be7, d0139f9f3e4e4235ac89980b174840fe; no voters:
I20260926 16:43:30.120502 12434 raft_consensus.cc:2781] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:30.121043 12434 raft_consensus.cc:686] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe [term 1 LEADER]: Becoming Leader. State: Replica: d0139f9f3e4e4235ac89980b174840fe, State: Running, Role: LEADER
I20260926 16:43:30.121904 12434 consensus_queue.cc:237] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } } peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } }
I20260926 16:43:30.126672 11919 catalog_manager.cc:5495] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe reported cstate change: term changed from 0 to 1, leader changed from <none> to d0139f9f3e4e4235ac89980b174840fe (127.9.238.194). New cstate: current_term: 1 leader_uuid: "d0139f9f3e4e4235ac89980b174840fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } health_report { overall_health: HEALTHY } } }
Sep 26 16:43:30 dist-test-slave-wt2j krb5kdc[11876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441004, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:30.220511 12083 raft_consensus.cc:2454] T 020d41b160b642d392e27503fb06c084 P 752d81f769294f1183734d69013940f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0139f9f3e4e4235ac89980b174840fe in term 1.
I20260926 16:43:30.224682 12296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:30.199135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54888 (local address 127.9.238.254:45877)
0926 16:43:30.199541 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:30.199551 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:30.200245 (+ 694us) server_negotiation.cc:407] Connection header received
0926 16:43:30.200496 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:30.200508 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:30.200572 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:30.200692 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:30.202335 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.203296 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.205019 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.205575 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.205755 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:30.216507 (+ 10752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:30.216513 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:30.216517 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:30.216560 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:30.220052 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:30.221667 (+ 1615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:30.221672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:30.221675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:30.222023 (+ 348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:30.222597 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:30.222601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:30.222603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:30.222993 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:30.223132 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:30.223942 (+ 810us) server_negotiation.cc:299] Negotiation successful
0926 16:43:30.224332 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":153}
I20260926 16:43:30.224879 12445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:30.199141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:45877 (local address 127.0.0.1:54888)
0926 16:43:30.200083 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:30.200103 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:30.200348 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:30.200735 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:30.200740 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:30.201769 (+ 1029us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:30.202134 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:30.202146 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.203446 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.203455 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:30.204832 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:30.204851 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.205817 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.205824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:30.206013 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:30.212391 (+ 6378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:30.212431 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:30.216177 (+ 3746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:30.220392 (+ 4215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:30.220418 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:30.220433 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:30.221451 (+ 1018us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:30.222172 (+ 721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:30.222180 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:30.222188 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:30.222426 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:30.223149 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:30.223162 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:30.223457 (+ 295us) client_negotiation.cc:769] Sending connection context
0926 16:43:30.223682 (+ 225us) client_negotiation.cc:240] Negotiation successful
0926 16:43:30.224143 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"thread_start_us":552,"threads_started":1}
I20260926 16:43:30.231895 11919 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:54888:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 16:43:30.233321 11919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:43:30.243273 12338 tablet_service.cc:1467] Processing CreateTablet for tablet 8dc08c22de6343bfb1dd429370c0cec3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c5b79e5b138740f6ba0a4619b2a9f679]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:43:30.243185 12198 tablet_service.cc:1467] Processing CreateTablet for tablet 8dc08c22de6343bfb1dd429370c0cec3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c5b79e5b138740f6ba0a4619b2a9f679]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:43:30.243185 12058 tablet_service.cc:1467] Processing CreateTablet for tablet 8dc08c22de6343bfb1dd429370c0cec3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c5b79e5b138740f6ba0a4619b2a9f679]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:43:30.244311 12338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dc08c22de6343bfb1dd429370c0cec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:30.244601 12198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dc08c22de6343bfb1dd429370c0cec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:30.244602 12058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dc08c22de6343bfb1dd429370c0cec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:30.250941 12429 tablet_bootstrap.cc:492] T 8dc08c22de6343bfb1dd429370c0cec3 P 7ce8ee95d02a4b278bb5a80c9e072be7: Bootstrap starting.
I20260926 16:43:30.252743 12429 tablet_bootstrap.cc:654] T 8dc08c22de6343bfb1dd429370c0cec3 P 7ce8ee95d02a4b278bb5a80c9e072be7: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:30.254050 12427 tablet_bootstrap.cc:492] T 8dc08c22de6343bfb1dd429370c0cec3 P d0139f9f3e4e4235ac89980b174840fe: Bootstrap starting.
I20260926 16:43:30.254799 12428 tablet_bootstrap.cc:492] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2: Bootstrap starting.
I20260926 16:43:30.255018 12429 tablet_bootstrap.cc:492] T 8dc08c22de6343bfb1dd429370c0cec3 P 7ce8ee95d02a4b278bb5a80c9e072be7: No bootstrap required, opened a new log
I20260926 16:43:30.255143 12429 ts_tablet_manager.cc:1397] T 8dc08c22de6343bfb1dd429370c0cec3 P 7ce8ee95d02a4b278bb5a80c9e072be7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 16:43:30.255931 12429 raft_consensus.cc:348] T 8dc08c22de6343bfb1dd429370c0cec3 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:30.256198 12429 raft_consensus.cc:374] T 8dc08c22de6343bfb1dd429370c0cec3 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:30.256289 12429 raft_consensus.cc:729] T 8dc08c22de6343bfb1dd429370c0cec3 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ce8ee95d02a4b278bb5a80c9e072be7, State: Initialized, Role: FOLLOWER
I20260926 16:43:30.256283 12427 tablet_bootstrap.cc:654] T 8dc08c22de6343bfb1dd429370c0cec3 P d0139f9f3e4e4235ac89980b174840fe: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:30.256559 12429 consensus_queue.cc:260] T 8dc08c22de6343bfb1dd429370c0cec3 P 7ce8ee95d02a4b278bb5a80c9e072be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:30.256810 12428 tablet_bootstrap.cc:654] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:30.257128 12433 tablet_replica.cc:439] T 8dc08c22de6343bfb1dd429370c0cec3 P 7ce8ee95d02a4b278bb5a80c9e072be7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } } }
I20260926 16:43:30.257411 12433 tablet_replica.cc:442] T 8dc08c22de6343bfb1dd429370c0cec3 P 7ce8ee95d02a4b278bb5a80c9e072be7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:43:30.259189 12429 ts_tablet_manager.cc:1428] T 8dc08c22de6343bfb1dd429370c0cec3 P 7ce8ee95d02a4b278bb5a80c9e072be7: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 16:43:30.260782 12428 tablet_bootstrap.cc:492] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2: No bootstrap required, opened a new log
I20260926 16:43:30.261021 12428 ts_tablet_manager.cc:1397] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 16:43:30.261958 12427 tablet_bootstrap.cc:492] T 8dc08c22de6343bfb1dd429370c0cec3 P d0139f9f3e4e4235ac89980b174840fe: No bootstrap required, opened a new log
I20260926 16:43:30.262166 12427 ts_tablet_manager.cc:1397] T 8dc08c22de6343bfb1dd429370c0cec3 P d0139f9f3e4e4235ac89980b174840fe: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20260926 16:43:30.262104 12428 raft_consensus.cc:348] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:30.262410 12428 raft_consensus.cc:374] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:30.262527 12428 raft_consensus.cc:729] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 752d81f769294f1183734d69013940f2, State: Initialized, Role: FOLLOWER
I20260926 16:43:30.262844 12428 consensus_queue.cc:260] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:30.263476 12435 tablet_replica.cc:439] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } } }
I20260926 16:43:30.263845 12427 raft_consensus.cc:348] T 8dc08c22de6343bfb1dd429370c0cec3 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:30.264076 12435 tablet_replica.cc:442] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:43:30.264148 12427 raft_consensus.cc:374] T 8dc08c22de6343bfb1dd429370c0cec3 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:30.264243 12427 raft_consensus.cc:729] T 8dc08c22de6343bfb1dd429370c0cec3 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0139f9f3e4e4235ac89980b174840fe, State: Initialized, Role: FOLLOWER
I20260926 16:43:30.264652 12428 ts_tablet_manager.cc:1428] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 16:43:30.264520 12427 consensus_queue.cc:260] T 8dc08c22de6343bfb1dd429370c0cec3 P d0139f9f3e4e4235ac89980b174840fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:30.265272 12427 ts_tablet_manager.cc:1428] T 8dc08c22de6343bfb1dd429370c0cec3 P d0139f9f3e4e4235ac89980b174840fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 16:43:30.265172 12434 tablet_replica.cc:439] T 8dc08c22de6343bfb1dd429370c0cec3 P d0139f9f3e4e4235ac89980b174840fe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } } }
I20260926 16:43:30.265602 12434 tablet_replica.cc:442] T 8dc08c22de6343bfb1dd429370c0cec3 P d0139f9f3e4e4235ac89980b174840fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:43:30.437767 12435 raft_consensus.cc:482] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:43:30.438014 12435 raft_consensus.cc:504] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:30.440042 12435 leader_election.cc:288] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0139f9f3e4e4235ac89980b174840fe (127.9.238.194:36687), 7ce8ee95d02a4b278bb5a80c9e072be7 (127.9.238.195:44787)
I20260926 16:43:30.448088 12449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:30.440478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:44787 (local address 127.9.238.193:33441)
0926 16:43:30.441073 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:30.441097 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:30.441307 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:30.441860 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:30.441864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:30.441888 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:30.442323 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:30.442337 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.443630 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.443636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:30.445702 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:30.445722 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.447116 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.447124 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:30.447332 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:30.447386 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 16:43:30.447514 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 16:43:30.447777 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":359,"threads_started":1}
I20260926 16:43:30.448153 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:30.440594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:33441 (local address 127.9.238.195:44787)
0926 16:43:30.440828 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:30.440837 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:30.441194 (+ 357us) server_negotiation.cc:407] Connection header received
0926 16:43:30.441481 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:30.441487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:30.441568 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:30.441641 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:30.442530 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.443436 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.445902 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.446821 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.447004 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:30.447074 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:30.447620 (+ 546us) server_negotiation.cc:299] Negotiation successful
0926 16:43:30.447783 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 16:43:30.449084 12358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dc08c22de6343bfb1dd429370c0cec3" candidate_uuid: "752d81f769294f1183734d69013940f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" is_pre_election: true
I20260926 16:43:30.449468 12358 raft_consensus.cc:2454] T 8dc08c22de6343bfb1dd429370c0cec3 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 752d81f769294f1183734d69013940f2 in term 0.
I20260926 16:43:30.450055 12450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:30.440496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:36687 (local address 127.9.238.193:58685)
0926 16:43:30.443370 (+ 2874us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:30.443389 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:30.443611 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:30.443980 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:30.443985 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:30.444017 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:30.444533 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:30.444553 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.445723 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.445729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:30.447422 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:30.447435 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.449113 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.449121 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:30.449313 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:30.449347 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 16:43:30.449474 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 16:43:30.449651 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2672,"thread_start_us":2387,"threads_started":1}
I20260926 16:43:30.450155 12009 leader_election.cc:302] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 752d81f769294f1183734d69013940f2, 7ce8ee95d02a4b278bb5a80c9e072be7; no voters:
I20260926 16:43:30.450738 12435 raft_consensus.cc:2781] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:43:30.450897 12451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:30.440610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:58685 (local address 127.9.238.194:36687)
0926 16:43:30.442777 (+ 2167us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:30.442784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:30.443511 (+ 727us) server_negotiation.cc:407] Connection header received
0926 16:43:30.443768 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:30.443771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:30.443834 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:30.443923 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:30.444763 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.445557 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.447646 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.448869 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.449074 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:30.449133 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:30.449521 (+ 388us) server_negotiation.cc:299] Negotiation successful
0926 16:43:30.449742 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2043,"thread_start_us":1920,"threads_started":1}
I20260926 16:43:30.450906 12435 raft_consensus.cc:482] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:43:30.451038 12435 raft_consensus.cc:3037] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:30.452099 12222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dc08c22de6343bfb1dd429370c0cec3" candidate_uuid: "752d81f769294f1183734d69013940f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0139f9f3e4e4235ac89980b174840fe" is_pre_election: true
I20260926 16:43:30.452550 12222 raft_consensus.cc:2454] T 8dc08c22de6343bfb1dd429370c0cec3 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 752d81f769294f1183734d69013940f2 in term 0.
I20260926 16:43:30.524271 12435 raft_consensus.cc:504] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:30.525133 12435 leader_election.cc:288] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2 [CANDIDATE]: Term 1 election: Requested vote from peers d0139f9f3e4e4235ac89980b174840fe (127.9.238.194:36687), 7ce8ee95d02a4b278bb5a80c9e072be7 (127.9.238.195:44787)
I20260926 16:43:30.526013 12358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dc08c22de6343bfb1dd429370c0cec3" candidate_uuid: "752d81f769294f1183734d69013940f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7"
I20260926 16:43:30.526201 12222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dc08c22de6343bfb1dd429370c0cec3" candidate_uuid: "752d81f769294f1183734d69013940f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0139f9f3e4e4235ac89980b174840fe"
I20260926 16:43:30.526371 12358 raft_consensus.cc:3037] T 8dc08c22de6343bfb1dd429370c0cec3 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:30.526515 12222 raft_consensus.cc:3037] T 8dc08c22de6343bfb1dd429370c0cec3 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:30.529042 12222 raft_consensus.cc:2454] T 8dc08c22de6343bfb1dd429370c0cec3 P d0139f9f3e4e4235ac89980b174840fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 752d81f769294f1183734d69013940f2 in term 1.
I20260926 16:43:30.529290 12358 raft_consensus.cc:2454] T 8dc08c22de6343bfb1dd429370c0cec3 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 752d81f769294f1183734d69013940f2 in term 1.
I20260926 16:43:30.529750 12010 leader_election.cc:302] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 752d81f769294f1183734d69013940f2, d0139f9f3e4e4235ac89980b174840fe; no voters:
I20260926 16:43:30.530519 12435 raft_consensus.cc:2781] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:30.531109 12435 raft_consensus.cc:686] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2 [term 1 LEADER]: Becoming Leader. State: Replica: 752d81f769294f1183734d69013940f2, State: Running, Role: LEADER
I20260926 16:43:30.531754 12435 consensus_queue.cc:237] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:30.535481 12452 tablet_replica.cc:439] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2: TxnStatusTablet state changed. Reason: New leader 752d81f769294f1183734d69013940f2. Latest consensus state: current_term: 1 leader_uuid: "752d81f769294f1183734d69013940f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } } }
I20260926 16:43:30.536209 12452 tablet_replica.cc:442] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:43:30.537444 12454 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 16:43:30.537778 12454 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 16:43:30.538467 12434 consensus_queue.cc:1035] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:43:30.538331 11920 catalog_manager.cc:5495] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 752d81f769294f1183734d69013940f2 (127.9.238.193). New cstate: current_term: 1 leader_uuid: "752d81f769294f1183734d69013940f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } health_report { overall_health: UNKNOWN } } }
I20260926 16:43:30.554477 12438 consensus_queue.cc:1035] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:43:30.568120 12445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:30.556897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:34407 (local address 127.0.0.1:47534)
0926 16:43:30.557258 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:30.557282 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:30.557448 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:30.558990 (+ 1542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:30.559000 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:30.559023 (+ 23us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:43:30.559350 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:30.559367 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.561873 (+ 2506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.561881 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:30.563607 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:30.563634 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.565360 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.565368 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:30.565581 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:30.565683 (+ 102us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:43:30.567303 (+ 1620us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:43:30.567332 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 16:43:30.567525 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 16:43:30.567773 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260926 16:43:30.568166 12426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:30.557801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47534 (local address 127.9.238.193:34407)
0926 16:43:30.558054 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:30.558060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:30.558081 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:43:30.558190 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:30.558195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:30.558305 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:30.558472 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:43:30.559792 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.561636 (+ 1844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.564365 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.565115 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.565309 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:30.566003 (+ 694us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:43:30.566990 (+ 987us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:43:30.567161 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:30.567593 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 16:43:30.567708 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 16:43:30.577883 12358 raft_consensus.cc:1261] T 8dc08c22de6343bfb1dd429370c0cec3 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 1 FOLLOWER]: Refusing update from remote peer 752d81f769294f1183734d69013940f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:43:30.578224 12222 raft_consensus.cc:1261] T 8dc08c22de6343bfb1dd429370c0cec3 P d0139f9f3e4e4235ac89980b174840fe [term 1 FOLLOWER]: Refusing update from remote peer 752d81f769294f1183734d69013940f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:43:30.578738 12452 consensus_queue.cc:1035] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:43:30.579226 12435 consensus_queue.cc:1035] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:43:30.588764 12438 tablet_replica.cc:439] T 8dc08c22de6343bfb1dd429370c0cec3 P d0139f9f3e4e4235ac89980b174840fe: TxnStatusTablet state changed. Reason: New leader 752d81f769294f1183734d69013940f2. Latest consensus state: current_term: 1 leader_uuid: "752d81f769294f1183734d69013940f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } } }
I20260926 16:43:30.589231 12438 tablet_replica.cc:442] T 8dc08c22de6343bfb1dd429370c0cec3 P d0139f9f3e4e4235ac89980b174840fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:43:30.591063 12433 tablet_replica.cc:439] T 8dc08c22de6343bfb1dd429370c0cec3 P 7ce8ee95d02a4b278bb5a80c9e072be7: TxnStatusTablet state changed. Reason: New leader 752d81f769294f1183734d69013940f2. Latest consensus state: current_term: 1 leader_uuid: "752d81f769294f1183734d69013940f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } } }
I20260926 16:43:30.591430 12433 tablet_replica.cc:442] T 8dc08c22de6343bfb1dd429370c0cec3 P 7ce8ee95d02a4b278bb5a80c9e072be7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:43:30.594882 12458 mvcc.cc:205] Tried to move back new op lower bound from 7333646379315769344 to 7333646379142381568. Current Snapshot: MvccSnapshot[applied={T|T < 7333646379315769344}]
I20260926 16:43:30.597406 12457 mvcc.cc:205] Tried to move back new op lower bound from 7333646379315769344 to 7333646379142381568. Current Snapshot: MvccSnapshot[applied={T|T < 7333646379315769344}]
I20260926 16:43:30.607244 12464 tablet_replica.cc:439] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "752d81f769294f1183734d69013940f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } } }
I20260926 16:43:30.607717 12464 tablet_replica.cc:442] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:43:30.607335 12438 tablet_replica.cc:439] T 8dc08c22de6343bfb1dd429370c0cec3 P d0139f9f3e4e4235ac89980b174840fe: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "752d81f769294f1183734d69013940f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } } }
I20260926 16:43:30.608536 12438 tablet_replica.cc:442] T 8dc08c22de6343bfb1dd429370c0cec3 P d0139f9f3e4e4235ac89980b174840fe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:43:30.613876 12464 tablet_replica.cc:439] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "752d81f769294f1183734d69013940f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } } }
I20260926 16:43:30.614179 12464 tablet_replica.cc:442] T 8dc08c22de6343bfb1dd429370c0cec3 P 752d81f769294f1183734d69013940f2: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:43:30.615061 12433 tablet_replica.cc:439] T 8dc08c22de6343bfb1dd429370c0cec3 P 7ce8ee95d02a4b278bb5a80c9e072be7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "752d81f769294f1183734d69013940f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } } }
I20260926 16:43:30.615507 12433 tablet_replica.cc:442] T 8dc08c22de6343bfb1dd429370c0cec3 P 7ce8ee95d02a4b278bb5a80c9e072be7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:43:30.626839 12451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:30.619407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39218 (local address 127.9.238.194:36687)
0926 16:43:30.619602 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:30.619609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:30.620316 (+ 707us) server_negotiation.cc:407] Connection header received
0926 16:43:30.620650 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:30.620655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:30.620763 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:30.621029 (+ 266us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:43:30.621502 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.622469 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.624114 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.624608 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.624760 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:30.625279 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:43:30.626050 (+ 771us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:43:30.626234 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:30.626322 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 16:43:30.626442 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 16:43:30.649061 12426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:30.641685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:33291 (local address 127.9.238.193:34407)
0926 16:43:30.641938 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:30.641945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:30.642481 (+ 536us) server_negotiation.cc:407] Connection header received
0926 16:43:30.642694 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:30.642700 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:30.642820 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:30.642979 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:43:30.643686 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.644889 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.646604 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.647300 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.647463 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:30.648166 (+ 703us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:43:30.648324 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:43:30.648427 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:30.648619 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 16:43:30.648724 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 16:43:30.649223 12472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:30.641462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:34407 (local address 127.9.238.194:33291)
0926 16:43:30.642308 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:30.642329 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:30.642534 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:30.643021 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:30.643025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:30.643037 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:43:30.643414 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:30.643425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.645085 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.645090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:30.646434 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:30.646448 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.647534 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.647540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:30.647774 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:30.647881 (+ 107us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:43:30.648452 (+ 571us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:43:30.648463 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 16:43:30.648576 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 16:43:30.648846 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":439,"threads_started":1}
I20260926 16:43:30.702622 12451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:30.693598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:56385 (local address 127.9.238.194:36687)
0926 16:43:30.693820 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:30.693828 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:30.694879 (+ 1051us) server_negotiation.cc:407] Connection header received
0926 16:43:30.695163 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:30.695169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:30.695282 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:30.695398 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:43:30.696397 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.697299 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.699804 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.700551 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.700746 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:30.701503 (+ 757us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:43:30.701697 (+ 194us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:43:30.701813 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:30.702082 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 16:43:30.702196 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 16:43:30.702872 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:30.693883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:36687 (local address 127.9.238.193:56385)
0926 16:43:30.694739 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:30.694765 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:30.694993 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:30.695663 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:30.695668 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:30.695679 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:43:30.696210 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:30.696231 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.697487 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.697492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:30.699475 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:30.699495 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.700813 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.700819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:30.700980 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:30.701088 (+ 108us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:43:30.701874 (+ 786us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:43:30.701912 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 16:43:30.702013 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 16:43:30.702220 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":415,"threads_started":1}
I20260926 16:43:30.761204 11920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:43:30.761632 11920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:43:30.767629 11920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77c9e7c806ed4ecd91a120bde5c165ef: Sending DeleteTablet for 3 replicas of tablet 020d41b160b642d392e27503fb06c084
I20260926 16:43:30.769274 12338 tablet_service.cc:1514] Processing DeleteTablet for tablet 020d41b160b642d392e27503fb06c084 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:30 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:37022
I20260926 16:43:30.771363 12058 tablet_service.cc:1514] Processing DeleteTablet for tablet 020d41b160b642d392e27503fb06c084 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:30 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:47518
I20260926 16:43:30.779242 12479 tablet_replica.cc:330] T 020d41b160b642d392e27503fb06c084 P 7ce8ee95d02a4b278bb5a80c9e072be7: stopping tablet replica
I20260926 16:43:30.779843 12479 raft_consensus.cc:2229] T 020d41b160b642d392e27503fb06c084 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:43:30.780272 12479 raft_consensus.cc:2258] T 020d41b160b642d392e27503fb06c084 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:43:30.780325 12198 tablet_service.cc:1514] Processing DeleteTablet for tablet 020d41b160b642d392e27503fb06c084 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:30 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:39210
I20260926 16:43:30.782531 12479 ts_tablet_manager.cc:1905] T 020d41b160b642d392e27503fb06c084 P 7ce8ee95d02a4b278bb5a80c9e072be7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:43:30.785514 11920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:43:30.786460 11920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:43:30.790747 12479 ts_tablet_manager.cc:1918] T 020d41b160b642d392e27503fb06c084 P 7ce8ee95d02a4b278bb5a80c9e072be7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:43:30.791054 12479 log.cc:1196] T 020d41b160b642d392e27503fb06c084 P 7ce8ee95d02a4b278bb5a80c9e072be7: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-2/wal/wals/020d41b160b642d392e27503fb06c084
I20260926 16:43:30.791707 12479 ts_tablet_manager.cc:1939] T 020d41b160b642d392e27503fb06c084 P 7ce8ee95d02a4b278bb5a80c9e072be7: Deleting consensus metadata
I20260926 16:43:30.794054 11906 catalog_manager.cc:4841] TS 7ce8ee95d02a4b278bb5a80c9e072be7 (127.9.238.195:44787): tablet 020d41b160b642d392e27503fb06c084 (table test-table [id=e793b92b7887404c86c8a34b125154ac]) successfully deleted
I20260926 16:43:30.796149 12481 tablet_replica.cc:330] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe: stopping tablet replica
I20260926 16:43:30.796813 12481 raft_consensus.cc:2229] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:43:30.797571 12481 raft_consensus.cc:2258] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:43:30.801288 12481 ts_tablet_manager.cc:1905] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:43:30.801250 12057 tablet_service.cc:1467] Processing CreateTablet for tablet a05bd90bdbcf4ebaa8c6151b700b3d89 (DEFAULT_TABLE table=test-table [id=3af3854167fc49dbb0a2213d603ffcb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:30.801815 12057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a05bd90bdbcf4ebaa8c6151b700b3d89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:30.802049 12338 tablet_service.cc:1467] Processing CreateTablet for tablet a05bd90bdbcf4ebaa8c6151b700b3d89 (DEFAULT_TABLE table=test-table [id=3af3854167fc49dbb0a2213d603ffcb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:30.802600 12338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a05bd90bdbcf4ebaa8c6151b700b3d89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:30.803251 12198 tablet_service.cc:1467] Processing CreateTablet for tablet a05bd90bdbcf4ebaa8c6151b700b3d89 (DEFAULT_TABLE table=test-table [id=3af3854167fc49dbb0a2213d603ffcb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:30.803778 12198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a05bd90bdbcf4ebaa8c6151b700b3d89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:30.822345 12481 ts_tablet_manager.cc:1918] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:43:30.822616 12481 log.cc:1196] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-1/wal/wals/020d41b160b642d392e27503fb06c084
I20260926 16:43:30.823243 12481 ts_tablet_manager.cc:1939] T 020d41b160b642d392e27503fb06c084 P d0139f9f3e4e4235ac89980b174840fe: Deleting consensus metadata
I20260926 16:43:30.825585 11907 catalog_manager.cc:4841] TS d0139f9f3e4e4235ac89980b174840fe (127.9.238.194:36687): tablet 020d41b160b642d392e27503fb06c084 (table test-table [id=e793b92b7887404c86c8a34b125154ac]) successfully deleted
I20260926 16:43:30.827436 12480 tablet_replica.cc:330] T 020d41b160b642d392e27503fb06c084 P 752d81f769294f1183734d69013940f2: stopping tablet replica
I20260926 16:43:30.830852 12480 raft_consensus.cc:2229] T 020d41b160b642d392e27503fb06c084 P 752d81f769294f1183734d69013940f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:43:30.831357 12480 raft_consensus.cc:2258] T 020d41b160b642d392e27503fb06c084 P 752d81f769294f1183734d69013940f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:43:30.836279 12483 tablet_bootstrap.cc:492] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7: Bootstrap starting.
I20260926 16:43:30.837502 12480 ts_tablet_manager.cc:1905] T 020d41b160b642d392e27503fb06c084 P 752d81f769294f1183734d69013940f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:43:30.838511 12483 tablet_bootstrap.cc:654] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:30.841727 12483 tablet_bootstrap.cc:492] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7: No bootstrap required, opened a new log
I20260926 16:43:30.841706 12484 tablet_bootstrap.cc:492] T a05bd90bdbcf4ebaa8c6151b700b3d89 P d0139f9f3e4e4235ac89980b174840fe: Bootstrap starting.
I20260926 16:43:30.841904 12483 ts_tablet_manager.cc:1397] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 16:43:30.842842 12483 raft_consensus.cc:348] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:30.843189 12483 raft_consensus.cc:374] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:30.843294 12483 raft_consensus.cc:729] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ce8ee95d02a4b278bb5a80c9e072be7, State: Initialized, Role: FOLLOWER
I20260926 16:43:30.843633 12483 consensus_queue.cc:260] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:30.844173 12484 tablet_bootstrap.cc:654] T a05bd90bdbcf4ebaa8c6151b700b3d89 P d0139f9f3e4e4235ac89980b174840fe: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:30.844487 12483 ts_tablet_manager.cc:1428] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 16:43:30.856671 12482 tablet_bootstrap.cc:492] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 752d81f769294f1183734d69013940f2: Bootstrap starting.
I20260926 16:43:30.858906 12482 tablet_bootstrap.cc:654] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 752d81f769294f1183734d69013940f2: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:30.861191 12484 tablet_bootstrap.cc:492] T a05bd90bdbcf4ebaa8c6151b700b3d89 P d0139f9f3e4e4235ac89980b174840fe: No bootstrap required, opened a new log
I20260926 16:43:30.861478 12484 ts_tablet_manager.cc:1397] T a05bd90bdbcf4ebaa8c6151b700b3d89 P d0139f9f3e4e4235ac89980b174840fe: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20260926 16:43:30.862628 12484 raft_consensus.cc:348] T a05bd90bdbcf4ebaa8c6151b700b3d89 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:30.862975 12484 raft_consensus.cc:374] T a05bd90bdbcf4ebaa8c6151b700b3d89 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:30.863112 12484 raft_consensus.cc:729] T a05bd90bdbcf4ebaa8c6151b700b3d89 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0139f9f3e4e4235ac89980b174840fe, State: Initialized, Role: FOLLOWER
I20260926 16:43:30.863494 12484 consensus_queue.cc:260] T a05bd90bdbcf4ebaa8c6151b700b3d89 P d0139f9f3e4e4235ac89980b174840fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:30.869831 12484 ts_tablet_manager.cc:1428] T a05bd90bdbcf4ebaa8c6151b700b3d89 P d0139f9f3e4e4235ac89980b174840fe: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20260926 16:43:30.873871 12480 ts_tablet_manager.cc:1918] T 020d41b160b642d392e27503fb06c084 P 752d81f769294f1183734d69013940f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:43:30.874168 12480 log.cc:1196] T 020d41b160b642d392e27503fb06c084 P 752d81f769294f1183734d69013940f2: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-0/wal/wals/020d41b160b642d392e27503fb06c084
I20260926 16:43:30.874861 12480 ts_tablet_manager.cc:1939] T 020d41b160b642d392e27503fb06c084 P 752d81f769294f1183734d69013940f2: Deleting consensus metadata
I20260926 16:43:30.880970 12482 tablet_bootstrap.cc:492] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 752d81f769294f1183734d69013940f2: No bootstrap required, opened a new log
I20260926 16:43:30.881278 12482 ts_tablet_manager.cc:1397] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 752d81f769294f1183734d69013940f2: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.000s
I20260926 16:43:30.882288 12482 raft_consensus.cc:348] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:30.882666 12482 raft_consensus.cc:374] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:30.882803 12482 raft_consensus.cc:729] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 752d81f769294f1183734d69013940f2, State: Initialized, Role: FOLLOWER
I20260926 16:43:30.883097 12482 consensus_queue.cc:260] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 752d81f769294f1183734d69013940f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:30.883169 11906 catalog_manager.cc:4841] TS 752d81f769294f1183734d69013940f2 (127.9.238.193:34407): tablet 020d41b160b642d392e27503fb06c084 (table test-table [id=e793b92b7887404c86c8a34b125154ac]) successfully deleted
I20260926 16:43:30.883949 12482 ts_tablet_manager.cc:1428] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 752d81f769294f1183734d69013940f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 16:43:30.891247 12433 raft_consensus.cc:482] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:43:30.891487 12433 raft_consensus.cc:504] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:30.893352 12433 leader_election.cc:288] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 752d81f769294f1183734d69013940f2 (127.9.238.193:34407), d0139f9f3e4e4235ac89980b174840fe (127.9.238.194:36687)
I20260926 16:43:30.913023 12426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:30.894405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:46563 (local address 127.9.238.193:34407)
0926 16:43:30.894890 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:30.894898 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:30.904272 (+ 9374us) server_negotiation.cc:407] Connection header received
0926 16:43:30.905189 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:30.905196 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:30.905276 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:30.905337 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:30.906234 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.907438 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.910199 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.911655 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.911819 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:30.911904 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:30.912541 (+ 637us) server_negotiation.cc:299] Negotiation successful
0926 16:43:30.912697 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318}
I20260926 16:43:30.913933 12489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:30.893716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:34407 (local address 127.9.238.195:46563)
0926 16:43:30.904117 (+ 10401us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:30.904142 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:30.904397 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:30.905546 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:30.905551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:30.905577 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:30.906038 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:30.906052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.907691 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.907696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:30.909983 (+ 2287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:30.909999 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.912132 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.912139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:30.912341 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:30.912381 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 16:43:30.913332 (+ 951us) client_negotiation.cc:240] Negotiation successful
0926 16:43:30.913497 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10217,"mutex_wait_us":10,"thread_start_us":10897,"threads_started":1}
I20260926 16:43:30.914790 12083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05bd90bdbcf4ebaa8c6151b700b3d89" candidate_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "752d81f769294f1183734d69013940f2" is_pre_election: true
I20260926 16:43:30.915210 12083 raft_consensus.cc:2454] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ce8ee95d02a4b278bb5a80c9e072be7 in term 0.
I20260926 16:43:30.916215 12289 leader_election.cc:302] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 752d81f769294f1183734d69013940f2, 7ce8ee95d02a4b278bb5a80c9e072be7; no voters:
I20260926 16:43:30.916949 12433 raft_consensus.cc:2781] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:43:30.917114 12433 raft_consensus.cc:482] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:43:30.917209 12433 raft_consensus.cc:3037] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:30.920405 12433 raft_consensus.cc:504] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:30.921217 12433 leader_election.cc:288] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7 [CANDIDATE]: Term 1 election: Requested vote from peers 752d81f769294f1183734d69013940f2 (127.9.238.193:34407), d0139f9f3e4e4235ac89980b174840fe (127.9.238.194:36687)
I20260926 16:43:30.921422 12451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:30.894643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:33041 (local address 127.9.238.194:36687)
0926 16:43:30.895059 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:30.895066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:30.904963 (+ 9897us) server_negotiation.cc:407] Connection header received
0926 16:43:30.905147 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:30.905156 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:30.905271 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:30.905436 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:30.909038 (+ 3602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.910440 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.917885 (+ 7445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.919666 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.919842 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:30.919946 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:30.920859 (+ 913us) server_negotiation.cc:299] Negotiation successful
0926 16:43:30.921039 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283}
I20260926 16:43:30.922153 12083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05bd90bdbcf4ebaa8c6151b700b3d89" candidate_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "752d81f769294f1183734d69013940f2"
I20260926 16:43:30.922531 12083 raft_consensus.cc:3037] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:30.927269 12488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:30.893941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:36687 (local address 127.9.238.195:33041)
0926 16:43:30.904849 (+ 10908us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:30.904866 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:30.904978 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:30.905649 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:30.905656 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:30.905727 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:30.908745 (+ 3018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:30.908764 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.910718 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.910725 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:30.916891 (+ 6166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:30.916912 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.920256 (+ 3344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.920264 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:30.920514 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:30.920559 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 16:43:30.926640 (+ 6081us) client_negotiation.cc:240] Negotiation successful
0926 16:43:30.926843 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10790,"mutex_wait_us":38,"thread_start_us":9818,"threads_started":1}
I20260926 16:43:30.927983 12083 raft_consensus.cc:2454] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 752d81f769294f1183734d69013940f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ce8ee95d02a4b278bb5a80c9e072be7 in term 1.
I20260926 16:43:30.928733 12289 leader_election.cc:302] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 752d81f769294f1183734d69013940f2, 7ce8ee95d02a4b278bb5a80c9e072be7; no voters:
I20260926 16:43:30.928619 12222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05bd90bdbcf4ebaa8c6151b700b3d89" candidate_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0139f9f3e4e4235ac89980b174840fe" is_pre_election: true
I20260926 16:43:30.928996 12222 raft_consensus.cc:2454] T a05bd90bdbcf4ebaa8c6151b700b3d89 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ce8ee95d02a4b278bb5a80c9e072be7 in term 0.
I20260926 16:43:30.929214 12433 raft_consensus.cc:2781] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:30.929458 12221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a05bd90bdbcf4ebaa8c6151b700b3d89" candidate_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0139f9f3e4e4235ac89980b174840fe"
I20260926 16:43:30.929780 12221 raft_consensus.cc:3037] T a05bd90bdbcf4ebaa8c6151b700b3d89 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:30.933997 12221 raft_consensus.cc:2454] T a05bd90bdbcf4ebaa8c6151b700b3d89 P d0139f9f3e4e4235ac89980b174840fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ce8ee95d02a4b278bb5a80c9e072be7 in term 1.
I20260926 16:43:30.936396 12433 raft_consensus.cc:686] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 1 LEADER]: Becoming Leader. State: Replica: 7ce8ee95d02a4b278bb5a80c9e072be7, State: Running, Role: LEADER
I20260926 16:43:30.937022 12433 consensus_queue.cc:237] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:30.943905 11920 catalog_manager.cc:5495] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ce8ee95d02a4b278bb5a80c9e072be7 (127.9.238.195). New cstate: current_term: 1 leader_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } health_report { overall_health: HEALTHY } } }
I20260926 16:43:30.990530 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:30.981575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37032 (local address 127.9.238.195:44787)
0926 16:43:30.982490 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:30.982497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:30.982523 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:43:30.982660 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:30.982665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:30.982803 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:30.983624 (+ 821us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:43:30.983800 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.985034 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.987035 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:30.987717 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:30.988339 (+ 622us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:30.988479 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:43:30.989295 (+ 816us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:43:30.989913 (+ 618us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:30.989971 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 16:43:30.990182 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":583,"threads_started":1}
I20260926 16:43:31.016541 12426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:31.007658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:41137 (local address 127.9.238.193:34407)
0926 16:43:31.007947 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:31.007955 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:31.008921 (+ 966us) server_negotiation.cc:407] Connection header received
0926 16:43:31.009192 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:31.009201 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:31.009318 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:31.009457 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:43:31.010139 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:31.011340 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:31.013879 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:31.014592 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:31.014765 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:31.015504 (+ 739us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:43:31.015742 (+ 238us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:43:31.015830 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:31.016052 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 16:43:31.016178 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260926 16:43:31.017923 12495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:31.007666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:34407 (local address 127.9.238.195:41137)
0926 16:43:31.008711 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:31.008744 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:31.009004 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:31.009507 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:31.009513 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:31.009540 (+ 27us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:43:31.009948 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:31.009961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:31.011532 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:31.011538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:31.013679 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:31.013698 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:31.014866 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:31.014873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:31.015118 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:31.015220 (+ 102us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:43:31.015913 (+ 693us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:43:31.015923 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 16:43:31.017240 (+ 1317us) client_negotiation.cc:240] Negotiation successful
0926 16:43:31.017462 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":554,"threads_started":1}
I20260926 16:43:31.035207 12222 raft_consensus.cc:1261] T a05bd90bdbcf4ebaa8c6151b700b3d89 P d0139f9f3e4e4235ac89980b174840fe [term 1 FOLLOWER]: Refusing update from remote peer 7ce8ee95d02a4b278bb5a80c9e072be7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:43:31.036309 12490 consensus_queue.cc:1035] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:43:31.041105 12083 raft_consensus.cc:1261] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 752d81f769294f1183734d69013940f2 [term 1 FOLLOWER]: Refusing update from remote peer 7ce8ee95d02a4b278bb5a80c9e072be7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:43:31.042514 12490 consensus_queue.cc:1035] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:43:31.055750 12461 mvcc.cc:205] Tried to move back new op lower bound from 7333646381190402048 to 7333646380803125248. Current Snapshot: MvccSnapshot[applied={T|T < 7333646381190402048}]
I20260926 16:43:31.093369 12475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:31.085743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:44787 (local address 127.9.238.193:33605)
0926 16:43:31.085984 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:31.086015 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:31.086158 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:31.086870 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:31.086876 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:31.086893 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:43:31.087150 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:31.087163 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:31.088566 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:31.088572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:31.090408 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:31.090428 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:31.091699 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:31.091705 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:31.091932 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:31.092009 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:43:31.092567 (+ 558us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:43:31.092586 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 16:43:31.092701 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 16:43:31.092904 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 16:43:31.094172 12492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:31.086126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:33605 (local address 127.9.238.195:44787)
0926 16:43:31.086393 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:31.086401 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:31.086436 (+ 35us) server_negotiation.cc:407] Connection header received
0926 16:43:31.086530 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:31.086536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:31.086639 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:31.086801 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:43:31.087354 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:31.088384 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:31.090656 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:31.091371 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:31.091523 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:31.092230 (+ 707us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:43:31.092408 (+ 178us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:43:31.092508 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:31.093738 (+ 1230us) server_negotiation.cc:299] Negotiation successful
0926 16:43:31.093861 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 16:43:31.095652 12461 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 16:43:32.747141 11920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:43:32.747638 11920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:43:32.752511 11920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77c9e7c806ed4ecd91a120bde5c165ef: Sending DeleteTablet for 3 replicas of tablet a05bd90bdbcf4ebaa8c6151b700b3d89
I20260926 16:43:32.753814 12198 tablet_service.cc:1514] Processing DeleteTablet for tablet a05bd90bdbcf4ebaa8c6151b700b3d89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:32 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:39210
I20260926 16:43:32.754194 12338 tablet_service.cc:1514] Processing DeleteTablet for tablet a05bd90bdbcf4ebaa8c6151b700b3d89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:32 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:37022
I20260926 16:43:32.754357 12057 tablet_service.cc:1514] Processing DeleteTablet for tablet a05bd90bdbcf4ebaa8c6151b700b3d89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:32 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:47518
I20260926 16:43:32.754695 12522 tablet_replica.cc:330] T a05bd90bdbcf4ebaa8c6151b700b3d89 P d0139f9f3e4e4235ac89980b174840fe: stopping tablet replica
I20260926 16:43:32.755321 12522 raft_consensus.cc:2229] T a05bd90bdbcf4ebaa8c6151b700b3d89 P d0139f9f3e4e4235ac89980b174840fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:43:32.755604 12522 raft_consensus.cc:2258] T a05bd90bdbcf4ebaa8c6151b700b3d89 P d0139f9f3e4e4235ac89980b174840fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:43:32.766312 12522 ts_tablet_manager.cc:1905] T a05bd90bdbcf4ebaa8c6151b700b3d89 P d0139f9f3e4e4235ac89980b174840fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:43:32.769770 12523 tablet_replica.cc:330] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7: stopping tablet replica
I20260926 16:43:32.772218 12523 raft_consensus.cc:2229] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:43:32.772825 12524 tablet_replica.cc:330] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 752d81f769294f1183734d69013940f2: stopping tablet replica
I20260926 16:43:32.772851 12523 raft_consensus.cc:2258] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:43:32.773124 12524 raft_consensus.cc:2229] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 752d81f769294f1183734d69013940f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:43:32.773413 12524 raft_consensus.cc:2258] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 752d81f769294f1183734d69013940f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:43:32.774132 12523 ts_tablet_manager.cc:1905] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:43:32.774726 12524 ts_tablet_manager.cc:1905] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 752d81f769294f1183734d69013940f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:43:32.778476 12522 ts_tablet_manager.cc:1918] T a05bd90bdbcf4ebaa8c6151b700b3d89 P d0139f9f3e4e4235ac89980b174840fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 16:43:32.778657 12522 log.cc:1196] T a05bd90bdbcf4ebaa8c6151b700b3d89 P d0139f9f3e4e4235ac89980b174840fe: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-1/wal/wals/a05bd90bdbcf4ebaa8c6151b700b3d89
I20260926 16:43:32.779206 12522 ts_tablet_manager.cc:1939] T a05bd90bdbcf4ebaa8c6151b700b3d89 P d0139f9f3e4e4235ac89980b174840fe: Deleting consensus metadata
I20260926 16:43:32.781663 12523 ts_tablet_manager.cc:1918] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 16:43:32.781936 12523 log.cc:1196] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-2/wal/wals/a05bd90bdbcf4ebaa8c6151b700b3d89
I20260926 16:43:32.782338 11907 catalog_manager.cc:4841] TS d0139f9f3e4e4235ac89980b174840fe (127.9.238.194:36687): tablet a05bd90bdbcf4ebaa8c6151b700b3d89 (table test-table [id=3af3854167fc49dbb0a2213d603ffcb1]) successfully deleted
I20260926 16:43:32.782546 12523 ts_tablet_manager.cc:1939] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 7ce8ee95d02a4b278bb5a80c9e072be7: Deleting consensus metadata
I20260926 16:43:32.783401 12524 ts_tablet_manager.cc:1918] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 752d81f769294f1183734d69013940f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 16:43:32.783617 12524 log.cc:1196] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 752d81f769294f1183734d69013940f2: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-0/wal/wals/a05bd90bdbcf4ebaa8c6151b700b3d89
I20260926 16:43:32.783649 11906 catalog_manager.cc:4841] TS 7ce8ee95d02a4b278bb5a80c9e072be7 (127.9.238.195:44787): tablet a05bd90bdbcf4ebaa8c6151b700b3d89 (table test-table [id=3af3854167fc49dbb0a2213d603ffcb1]) successfully deleted
I20260926 16:43:32.784158 12524 ts_tablet_manager.cc:1939] T a05bd90bdbcf4ebaa8c6151b700b3d89 P 752d81f769294f1183734d69013940f2: Deleting consensus metadata
I20260926 16:43:32.785609 11906 catalog_manager.cc:4841] TS 752d81f769294f1183734d69013940f2 (127.9.238.193:34407): tablet a05bd90bdbcf4ebaa8c6151b700b3d89 (table test-table [id=3af3854167fc49dbb0a2213d603ffcb1]) successfully deleted
Sep 26 16:43:32 dist-test-slave-wt2j krb5kdc[11876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441012, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 16:43:32 dist-test-slave-wt2j krb5kdc[11876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441012, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:32.830775 12532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:32.808813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44336 (local address 127.9.238.254:45877)
0926 16:43:32.812123 (+ 3310us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:32.812128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:32.812168 (+ 40us) server_negotiation.cc:407] Connection header received
0926 16:43:32.812289 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:32.812296 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:32.812389 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:32.812554 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:32.814180 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:32.815055 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:32.816771 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:32.817623 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:32.817937 (+ 314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:32.823990 (+ 6053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:32.823997 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:32.824001 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:32.824059 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:32.827424 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:32.828359 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:32.828368 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:32.828372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:32.828544 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:32.828995 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:32.829002 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:32.829006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:32.829346 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:32.829647 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:32.830015 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 16:43:32.830320 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3161,"thread_start_us":2960,"threads_started":1}
I20260926 16:43:32.835798 11920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:43:32.836633 11920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:43:32.844862 12057 tablet_service.cc:1467] Processing CreateTablet for tablet ff80dca148a44ea59d439e8e58d481b8 (DEFAULT_TABLE table=test-table [id=4ab3eef4b2a84ac18fe10f8e0e38f318]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:32.845033 12198 tablet_service.cc:1467] Processing CreateTablet for tablet ff80dca148a44ea59d439e8e58d481b8 (DEFAULT_TABLE table=test-table [id=4ab3eef4b2a84ac18fe10f8e0e38f318]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:32.845155 12338 tablet_service.cc:1467] Processing CreateTablet for tablet ff80dca148a44ea59d439e8e58d481b8 (DEFAULT_TABLE table=test-table [id=4ab3eef4b2a84ac18fe10f8e0e38f318]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:32.845494 12198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff80dca148a44ea59d439e8e58d481b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:32.845525 12057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff80dca148a44ea59d439e8e58d481b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:32.845666 12338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff80dca148a44ea59d439e8e58d481b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:32.852666 12533 tablet_bootstrap.cc:492] T ff80dca148a44ea59d439e8e58d481b8 P 7ce8ee95d02a4b278bb5a80c9e072be7: Bootstrap starting.
I20260926 16:43:32.854028 12534 tablet_bootstrap.cc:492] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2: Bootstrap starting.
I20260926 16:43:32.855180 12535 tablet_bootstrap.cc:492] T ff80dca148a44ea59d439e8e58d481b8 P d0139f9f3e4e4235ac89980b174840fe: Bootstrap starting.
I20260926 16:43:32.855943 12533 tablet_bootstrap.cc:654] T ff80dca148a44ea59d439e8e58d481b8 P 7ce8ee95d02a4b278bb5a80c9e072be7: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:32.855953 12534 tablet_bootstrap.cc:654] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:32.857110 12535 tablet_bootstrap.cc:654] T ff80dca148a44ea59d439e8e58d481b8 P d0139f9f3e4e4235ac89980b174840fe: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:32.859071 12534 tablet_bootstrap.cc:492] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2: No bootstrap required, opened a new log
I20260926 16:43:32.859809 12534 ts_tablet_manager.cc:1397] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 16:43:32.860014 12535 tablet_bootstrap.cc:492] T ff80dca148a44ea59d439e8e58d481b8 P d0139f9f3e4e4235ac89980b174840fe: No bootstrap required, opened a new log
I20260926 16:43:32.860193 12535 ts_tablet_manager.cc:1397] T ff80dca148a44ea59d439e8e58d481b8 P d0139f9f3e4e4235ac89980b174840fe: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 16:43:32.860720 12534 raft_consensus.cc:348] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:32.860993 12534 raft_consensus.cc:374] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:32.861106 12534 raft_consensus.cc:729] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 752d81f769294f1183734d69013940f2, State: Initialized, Role: FOLLOWER
I20260926 16:43:32.861070 12535 raft_consensus.cc:348] T ff80dca148a44ea59d439e8e58d481b8 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:32.861383 12535 raft_consensus.cc:374] T ff80dca148a44ea59d439e8e58d481b8 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:32.861502 12535 raft_consensus.cc:729] T ff80dca148a44ea59d439e8e58d481b8 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0139f9f3e4e4235ac89980b174840fe, State: Initialized, Role: FOLLOWER
I20260926 16:43:32.861404 12534 consensus_queue.cc:260] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:32.861778 12533 tablet_bootstrap.cc:492] T ff80dca148a44ea59d439e8e58d481b8 P 7ce8ee95d02a4b278bb5a80c9e072be7: No bootstrap required, opened a new log
I20260926 16:43:32.861809 12535 consensus_queue.cc:260] T ff80dca148a44ea59d439e8e58d481b8 P d0139f9f3e4e4235ac89980b174840fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:32.862007 12533 ts_tablet_manager.cc:1397] T ff80dca148a44ea59d439e8e58d481b8 P 7ce8ee95d02a4b278bb5a80c9e072be7: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260926 16:43:32.862129 12534 ts_tablet_manager.cc:1428] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 16:43:32.862860 12535 ts_tablet_manager.cc:1428] T ff80dca148a44ea59d439e8e58d481b8 P d0139f9f3e4e4235ac89980b174840fe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 16:43:32.863161 12533 raft_consensus.cc:348] T ff80dca148a44ea59d439e8e58d481b8 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:32.864557 12533 raft_consensus.cc:374] T ff80dca148a44ea59d439e8e58d481b8 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:32.864708 12533 raft_consensus.cc:729] T ff80dca148a44ea59d439e8e58d481b8 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ce8ee95d02a4b278bb5a80c9e072be7, State: Initialized, Role: FOLLOWER
I20260926 16:43:32.865063 12533 consensus_queue.cc:260] T ff80dca148a44ea59d439e8e58d481b8 P 7ce8ee95d02a4b278bb5a80c9e072be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:32.866144 12533 ts_tablet_manager.cc:1428] T ff80dca148a44ea59d439e8e58d481b8 P 7ce8ee95d02a4b278bb5a80c9e072be7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260926 16:43:33.125878 12477 raft_consensus.cc:482] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:43:33.126232 12477 raft_consensus.cc:504] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:33.127206 12477 leader_election.cc:288] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0139f9f3e4e4235ac89980b174840fe (127.9.238.194:36687), 7ce8ee95d02a4b278bb5a80c9e072be7 (127.9.238.195:44787)
I20260926 16:43:33.128124 12222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff80dca148a44ea59d439e8e58d481b8" candidate_uuid: "752d81f769294f1183734d69013940f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0139f9f3e4e4235ac89980b174840fe" is_pre_election: true
I20260926 16:43:33.128275 12358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff80dca148a44ea59d439e8e58d481b8" candidate_uuid: "752d81f769294f1183734d69013940f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" is_pre_election: true
I20260926 16:43:33.128469 12222 raft_consensus.cc:2454] T ff80dca148a44ea59d439e8e58d481b8 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 752d81f769294f1183734d69013940f2 in term 0.
I20260926 16:43:33.128693 12358 raft_consensus.cc:2454] T ff80dca148a44ea59d439e8e58d481b8 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 752d81f769294f1183734d69013940f2 in term 0.
I20260926 16:43:33.129092 12010 leader_election.cc:302] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 752d81f769294f1183734d69013940f2, d0139f9f3e4e4235ac89980b174840fe; no voters:
I20260926 16:43:33.129442 12477 raft_consensus.cc:2781] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:43:33.129596 12477 raft_consensus.cc:482] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:43:33.129743 12477 raft_consensus.cc:3037] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:33.132464 12477 raft_consensus.cc:504] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:33.133313 12477 leader_election.cc:288] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2 [CANDIDATE]: Term 1 election: Requested vote from peers d0139f9f3e4e4235ac89980b174840fe (127.9.238.194:36687), 7ce8ee95d02a4b278bb5a80c9e072be7 (127.9.238.195:44787)
I20260926 16:43:33.134032 12222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff80dca148a44ea59d439e8e58d481b8" candidate_uuid: "752d81f769294f1183734d69013940f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0139f9f3e4e4235ac89980b174840fe"
I20260926 16:43:33.134110 12358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff80dca148a44ea59d439e8e58d481b8" candidate_uuid: "752d81f769294f1183734d69013940f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7"
I20260926 16:43:33.134344 12222 raft_consensus.cc:3037] T ff80dca148a44ea59d439e8e58d481b8 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:33.134382 12358 raft_consensus.cc:3037] T ff80dca148a44ea59d439e8e58d481b8 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:33.136718 12358 raft_consensus.cc:2454] T ff80dca148a44ea59d439e8e58d481b8 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 752d81f769294f1183734d69013940f2 in term 1.
I20260926 16:43:33.137212 12009 leader_election.cc:302] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 752d81f769294f1183734d69013940f2, 7ce8ee95d02a4b278bb5a80c9e072be7; no voters:
I20260926 16:43:33.137519 12222 raft_consensus.cc:2454] T ff80dca148a44ea59d439e8e58d481b8 P d0139f9f3e4e4235ac89980b174840fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 752d81f769294f1183734d69013940f2 in term 1.
I20260926 16:43:33.137589 12477 raft_consensus.cc:2781] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:33.137744 12477 raft_consensus.cc:686] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2 [term 1 LEADER]: Becoming Leader. State: Replica: 752d81f769294f1183734d69013940f2, State: Running, Role: LEADER
I20260926 16:43:33.138345 12477 consensus_queue.cc:237] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:33.141943 11919 catalog_manager.cc:5495] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 752d81f769294f1183734d69013940f2 (127.9.238.193). New cstate: current_term: 1 leader_uuid: "752d81f769294f1183734d69013940f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } health_report { overall_health: UNKNOWN } } }
I20260926 16:43:33.182916 12543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:33.176618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49896 (local address 127.9.238.193:34407)
0926 16:43:33.177530 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:33.177535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:33.177562 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:43:33.177642 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:33.177647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:33.177790 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:33.177930 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:43:33.178529 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:33.179570 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:33.180966 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:33.181431 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:33.181542 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:33.182003 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:43:33.182138 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:43:33.182215 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:33.182375 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 16:43:33.182451 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":772,"thread_start_us":574,"threads_started":1}
I20260926 16:43:33.199359 12543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:33.193599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:56115 (local address 127.9.238.193:34407)
0926 16:43:33.193803 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:33.193808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:33.194087 (+ 279us) server_negotiation.cc:407] Connection header received
0926 16:43:33.194273 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:33.194277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:33.194357 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:33.194490 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:43:33.195005 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:33.195907 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:33.197413 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:33.197912 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:33.198052 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:33.198526 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:43:33.198662 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:43:33.198752 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:33.198971 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 16:43:33.199054 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 16:43:33.199534 12546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:33.193269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:34407 (local address 127.9.238.193:56115)
0926 16:43:33.193978 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:33.193995 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:33.194137 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:33.194565 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:33.194568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:33.194581 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:43:33.194817 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:33.194825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:33.196076 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:33.196080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:33.197221 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:33.197235 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:33.198134 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:33.198138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:33.198274 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:33.198324 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:43:33.198808 (+ 484us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:43:33.198820 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 16:43:33.198912 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 16:43:33.199041 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":422,"threads_started":1}
I20260926 16:43:33.209581 12358 raft_consensus.cc:1261] T ff80dca148a44ea59d439e8e58d481b8 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 1 FOLLOWER]: Refusing update from remote peer 752d81f769294f1183734d69013940f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:43:33.209542 12222 raft_consensus.cc:1261] T ff80dca148a44ea59d439e8e58d481b8 P d0139f9f3e4e4235ac89980b174840fe [term 1 FOLLOWER]: Refusing update from remote peer 752d81f769294f1183734d69013940f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:43:33.210356 12541 consensus_queue.cc:1035] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:43:33.210937 12503 consensus_queue.cc:1035] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:43:33.266963 11920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:43:33.267324 11920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:43:33.271121 11920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77c9e7c806ed4ecd91a120bde5c165ef: Sending DeleteTablet for 3 replicas of tablet ff80dca148a44ea59d439e8e58d481b8
I20260926 16:43:33.272351 12338 tablet_service.cc:1514] Processing DeleteTablet for tablet ff80dca148a44ea59d439e8e58d481b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:33 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:37022
I20260926 16:43:33.272440 12198 tablet_service.cc:1514] Processing DeleteTablet for tablet ff80dca148a44ea59d439e8e58d481b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:33 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:39210
I20260926 16:43:33.272722 12523 tablet_replica.cc:330] T ff80dca148a44ea59d439e8e58d481b8 P 7ce8ee95d02a4b278bb5a80c9e072be7: stopping tablet replica
I20260926 16:43:33.272756 12057 tablet_service.cc:1514] Processing DeleteTablet for tablet ff80dca148a44ea59d439e8e58d481b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:33 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:47518
I20260926 16:43:33.272935 12523 raft_consensus.cc:2229] T ff80dca148a44ea59d439e8e58d481b8 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:43:33.273133 12523 raft_consensus.cc:2258] T ff80dca148a44ea59d439e8e58d481b8 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:43:33.273144 12524 tablet_replica.cc:330] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2: stopping tablet replica
I20260926 16:43:33.273170 12522 tablet_replica.cc:330] T ff80dca148a44ea59d439e8e58d481b8 P d0139f9f3e4e4235ac89980b174840fe: stopping tablet replica
I20260926 16:43:33.273396 12522 raft_consensus.cc:2229] T ff80dca148a44ea59d439e8e58d481b8 P d0139f9f3e4e4235ac89980b174840fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:43:33.273432 12524 raft_consensus.cc:2229] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:43:33.273592 12522 raft_consensus.cc:2258] T ff80dca148a44ea59d439e8e58d481b8 P d0139f9f3e4e4235ac89980b174840fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:43:33.274308 12524 raft_consensus.cc:2258] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:43:33.274981 12522 ts_tablet_manager.cc:1905] T ff80dca148a44ea59d439e8e58d481b8 P d0139f9f3e4e4235ac89980b174840fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:43:33.275916 12524 ts_tablet_manager.cc:1905] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:43:33.280717 12523 ts_tablet_manager.cc:1905] T ff80dca148a44ea59d439e8e58d481b8 P 7ce8ee95d02a4b278bb5a80c9e072be7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:43:33.280534 11920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:43:33.281703 11920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:43:33.286634 12524 ts_tablet_manager.cc:1918] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:43:33.286888 12524 log.cc:1196] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-0/wal/wals/ff80dca148a44ea59d439e8e58d481b8
I20260926 16:43:33.288679 12524 ts_tablet_manager.cc:1939] T ff80dca148a44ea59d439e8e58d481b8 P 752d81f769294f1183734d69013940f2: Deleting consensus metadata
I20260926 16:43:33.290520 11906 catalog_manager.cc:4841] TS 752d81f769294f1183734d69013940f2 (127.9.238.193:34407): tablet ff80dca148a44ea59d439e8e58d481b8 (table test-table [id=4ab3eef4b2a84ac18fe10f8e0e38f318]) successfully deleted
I20260926 16:43:33.291209 12057 tablet_service.cc:1467] Processing CreateTablet for tablet 6bbf52826c42434cb40a0238f5d6bbb9 (DEFAULT_TABLE table=test-table [id=6c2e8df8ed0540a38d3c2e17ea5e1e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:33.291740 12057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bbf52826c42434cb40a0238f5d6bbb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:33.292661 12198 tablet_service.cc:1467] Processing CreateTablet for tablet 6bbf52826c42434cb40a0238f5d6bbb9 (DEFAULT_TABLE table=test-table [id=6c2e8df8ed0540a38d3c2e17ea5e1e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:33.293243 12198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bbf52826c42434cb40a0238f5d6bbb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:33.293380 12522 ts_tablet_manager.cc:1918] T ff80dca148a44ea59d439e8e58d481b8 P d0139f9f3e4e4235ac89980b174840fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:43:33.293606 12522 log.cc:1196] T ff80dca148a44ea59d439e8e58d481b8 P d0139f9f3e4e4235ac89980b174840fe: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-1/wal/wals/ff80dca148a44ea59d439e8e58d481b8
I20260926 16:43:33.294246 12522 ts_tablet_manager.cc:1939] T ff80dca148a44ea59d439e8e58d481b8 P d0139f9f3e4e4235ac89980b174840fe: Deleting consensus metadata
I20260926 16:43:33.295917 11907 catalog_manager.cc:4841] TS d0139f9f3e4e4235ac89980b174840fe (127.9.238.194:36687): tablet ff80dca148a44ea59d439e8e58d481b8 (table test-table [id=4ab3eef4b2a84ac18fe10f8e0e38f318]) successfully deleted
I20260926 16:43:33.291204 12338 tablet_service.cc:1467] Processing CreateTablet for tablet 6bbf52826c42434cb40a0238f5d6bbb9 (DEFAULT_TABLE table=test-table [id=6c2e8df8ed0540a38d3c2e17ea5e1e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:33.296911 12338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bbf52826c42434cb40a0238f5d6bbb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:33.299207 12534 tablet_bootstrap.cc:492] T 6bbf52826c42434cb40a0238f5d6bbb9 P 752d81f769294f1183734d69013940f2: Bootstrap starting.
I20260926 16:43:33.301231 12534 tablet_bootstrap.cc:654] T 6bbf52826c42434cb40a0238f5d6bbb9 P 752d81f769294f1183734d69013940f2: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:33.302301 12535 tablet_bootstrap.cc:492] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe: Bootstrap starting.
I20260926 16:43:33.288581 12523 ts_tablet_manager.cc:1918] T ff80dca148a44ea59d439e8e58d481b8 P 7ce8ee95d02a4b278bb5a80c9e072be7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:43:33.303815 12535 tablet_bootstrap.cc:654] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:33.304744 12523 log.cc:1196] T ff80dca148a44ea59d439e8e58d481b8 P 7ce8ee95d02a4b278bb5a80c9e072be7: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-2/wal/wals/ff80dca148a44ea59d439e8e58d481b8
I20260926 16:43:33.305529 12523 ts_tablet_manager.cc:1939] T ff80dca148a44ea59d439e8e58d481b8 P 7ce8ee95d02a4b278bb5a80c9e072be7: Deleting consensus metadata
I20260926 16:43:33.308084 11906 catalog_manager.cc:4841] TS 7ce8ee95d02a4b278bb5a80c9e072be7 (127.9.238.195:44787): tablet ff80dca148a44ea59d439e8e58d481b8 (table test-table [id=4ab3eef4b2a84ac18fe10f8e0e38f318]) successfully deleted
I20260926 16:43:33.308745 12534 tablet_bootstrap.cc:492] T 6bbf52826c42434cb40a0238f5d6bbb9 P 752d81f769294f1183734d69013940f2: No bootstrap required, opened a new log
I20260926 16:43:33.309115 12533 tablet_bootstrap.cc:492] T 6bbf52826c42434cb40a0238f5d6bbb9 P 7ce8ee95d02a4b278bb5a80c9e072be7: Bootstrap starting.
I20260926 16:43:33.310549 12533 tablet_bootstrap.cc:654] T 6bbf52826c42434cb40a0238f5d6bbb9 P 7ce8ee95d02a4b278bb5a80c9e072be7: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:33.311616 12534 ts_tablet_manager.cc:1397] T 6bbf52826c42434cb40a0238f5d6bbb9 P 752d81f769294f1183734d69013940f2: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260926 16:43:33.312577 12534 raft_consensus.cc:348] T 6bbf52826c42434cb40a0238f5d6bbb9 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:33.312824 12534 raft_consensus.cc:374] T 6bbf52826c42434cb40a0238f5d6bbb9 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:33.312943 12534 raft_consensus.cc:729] T 6bbf52826c42434cb40a0238f5d6bbb9 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 752d81f769294f1183734d69013940f2, State: Initialized, Role: FOLLOWER
I20260926 16:43:33.313244 12534 consensus_queue.cc:260] T 6bbf52826c42434cb40a0238f5d6bbb9 P 752d81f769294f1183734d69013940f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:33.313767 12534 ts_tablet_manager.cc:1428] T 6bbf52826c42434cb40a0238f5d6bbb9 P 752d81f769294f1183734d69013940f2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260926 16:43:33.315320 12533 tablet_bootstrap.cc:492] T 6bbf52826c42434cb40a0238f5d6bbb9 P 7ce8ee95d02a4b278bb5a80c9e072be7: No bootstrap required, opened a new log
I20260926 16:43:33.315503 12533 ts_tablet_manager.cc:1397] T 6bbf52826c42434cb40a0238f5d6bbb9 P 7ce8ee95d02a4b278bb5a80c9e072be7: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260926 16:43:33.316334 12533 raft_consensus.cc:348] T 6bbf52826c42434cb40a0238f5d6bbb9 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:33.316639 12533 raft_consensus.cc:374] T 6bbf52826c42434cb40a0238f5d6bbb9 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:33.316740 12533 raft_consensus.cc:729] T 6bbf52826c42434cb40a0238f5d6bbb9 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ce8ee95d02a4b278bb5a80c9e072be7, State: Initialized, Role: FOLLOWER
I20260926 16:43:33.317011 12533 consensus_queue.cc:260] T 6bbf52826c42434cb40a0238f5d6bbb9 P 7ce8ee95d02a4b278bb5a80c9e072be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:33.317749 12533 ts_tablet_manager.cc:1428] T 6bbf52826c42434cb40a0238f5d6bbb9 P 7ce8ee95d02a4b278bb5a80c9e072be7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 16:43:33.325428 12535 tablet_bootstrap.cc:492] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe: No bootstrap required, opened a new log
I20260926 16:43:33.325717 12535 ts_tablet_manager.cc:1397] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.009s
I20260926 16:43:33.326913 12535 raft_consensus.cc:348] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:33.327257 12535 raft_consensus.cc:374] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:33.327353 12535 raft_consensus.cc:729] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0139f9f3e4e4235ac89980b174840fe, State: Initialized, Role: FOLLOWER
I20260926 16:43:33.327716 12535 consensus_queue.cc:260] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:33.329259 12535 ts_tablet_manager.cc:1428] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 16:43:33.450318 12539 raft_consensus.cc:482] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:43:33.450706 12539 raft_consensus.cc:504] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:33.451742 12539 leader_election.cc:288] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 752d81f769294f1183734d69013940f2 (127.9.238.193:34407), 7ce8ee95d02a4b278bb5a80c9e072be7 (127.9.238.195:44787)
I20260926 16:43:33.452581 12358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bbf52826c42434cb40a0238f5d6bbb9" candidate_uuid: "d0139f9f3e4e4235ac89980b174840fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" is_pre_election: true
I20260926 16:43:33.452538 12083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bbf52826c42434cb40a0238f5d6bbb9" candidate_uuid: "d0139f9f3e4e4235ac89980b174840fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "752d81f769294f1183734d69013940f2" is_pre_election: true
I20260926 16:43:33.452920 12083 raft_consensus.cc:2454] T 6bbf52826c42434cb40a0238f5d6bbb9 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0139f9f3e4e4235ac89980b174840fe in term 0.
I20260926 16:43:33.452919 12358 raft_consensus.cc:2454] T 6bbf52826c42434cb40a0238f5d6bbb9 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0139f9f3e4e4235ac89980b174840fe in term 0.
I20260926 16:43:33.453464 12149 leader_election.cc:302] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ce8ee95d02a4b278bb5a80c9e072be7, d0139f9f3e4e4235ac89980b174840fe; no voters:
I20260926 16:43:33.453835 12539 raft_consensus.cc:2781] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:43:33.454035 12539 raft_consensus.cc:482] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:43:33.454125 12539 raft_consensus.cc:3037] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:33.457247 12539 raft_consensus.cc:504] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:33.458108 12539 leader_election.cc:288] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe [CANDIDATE]: Term 1 election: Requested vote from peers 752d81f769294f1183734d69013940f2 (127.9.238.193:34407), 7ce8ee95d02a4b278bb5a80c9e072be7 (127.9.238.195:44787)
I20260926 16:43:33.459137 12358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bbf52826c42434cb40a0238f5d6bbb9" candidate_uuid: "d0139f9f3e4e4235ac89980b174840fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7"
I20260926 16:43:33.459096 12083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bbf52826c42434cb40a0238f5d6bbb9" candidate_uuid: "d0139f9f3e4e4235ac89980b174840fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "752d81f769294f1183734d69013940f2"
I20260926 16:43:33.459455 12358 raft_consensus.cc:3037] T 6bbf52826c42434cb40a0238f5d6bbb9 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:33.459512 12083 raft_consensus.cc:3037] T 6bbf52826c42434cb40a0238f5d6bbb9 P 752d81f769294f1183734d69013940f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:33.463100 12358 raft_consensus.cc:2454] T 6bbf52826c42434cb40a0238f5d6bbb9 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0139f9f3e4e4235ac89980b174840fe in term 1.
I20260926 16:43:33.463135 12083 raft_consensus.cc:2454] T 6bbf52826c42434cb40a0238f5d6bbb9 P 752d81f769294f1183734d69013940f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0139f9f3e4e4235ac89980b174840fe in term 1.
I20260926 16:43:33.463816 12149 leader_election.cc:302] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ce8ee95d02a4b278bb5a80c9e072be7, d0139f9f3e4e4235ac89980b174840fe; no voters:
I20260926 16:43:33.464304 12539 raft_consensus.cc:2781] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:33.464895 12539 raft_consensus.cc:686] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe [term 1 LEADER]: Becoming Leader. State: Replica: d0139f9f3e4e4235ac89980b174840fe, State: Running, Role: LEADER
I20260926 16:43:33.465252 12539 consensus_queue.cc:237] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } }
I20260926 16:43:33.469233 11919 catalog_manager.cc:5495] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe reported cstate change: term changed from 0 to 1, leader changed from <none> to d0139f9f3e4e4235ac89980b174840fe (127.9.238.194). New cstate: current_term: 1 leader_uuid: "d0139f9f3e4e4235ac89980b174840fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0139f9f3e4e4235ac89980b174840fe" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 } health_report { overall_health: UNKNOWN } } }
I20260926 16:43:33.531430 12556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:33.524346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57606 (local address 127.9.238.194:36687)
0926 16:43:33.525096 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:33.525104 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:33.525134 (+ 30us) server_negotiation.cc:407] Connection header received
0926 16:43:33.525232 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:33.525236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:33.525316 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:33.525417 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:43:33.525936 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:33.526876 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:33.528946 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:33.529435 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:33.529558 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:33.530256 (+ 698us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:43:33.530420 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:43:33.530511 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:33.530729 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 16:43:33.530850 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":746,"threads_started":1}
I20260926 16:43:33.544752 12083 raft_consensus.cc:1261] T 6bbf52826c42434cb40a0238f5d6bbb9 P 752d81f769294f1183734d69013940f2 [term 1 FOLLOWER]: Refusing update from remote peer d0139f9f3e4e4235ac89980b174840fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:43:33.544725 12358 raft_consensus.cc:1261] T 6bbf52826c42434cb40a0238f5d6bbb9 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 1 FOLLOWER]: Refusing update from remote peer d0139f9f3e4e4235ac89980b174840fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:43:33.545414 12539 consensus_queue.cc:1035] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ce8ee95d02a4b278bb5a80c9e072be7" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:43:33.545787 12554 consensus_queue.cc:1035] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "752d81f769294f1183734d69013940f2" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 34407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 16:43:33.588603 12504 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 16:43:35.229048 11919 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:43:35.229341 11919 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:43:35.234198 11919 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77c9e7c806ed4ecd91a120bde5c165ef: Sending DeleteTablet for 3 replicas of tablet 6bbf52826c42434cb40a0238f5d6bbb9
I20260926 16:43:35.235200 12057 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bbf52826c42434cb40a0238f5d6bbb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:35 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:47518
I20260926 16:43:35.235286 12338 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bbf52826c42434cb40a0238f5d6bbb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:35 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:37022
I20260926 16:43:35.235373 12198 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bbf52826c42434cb40a0238f5d6bbb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:35 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:39210
I20260926 16:43:35.236341 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 11992
I20260926 16:43:35.247896 12581 tablet_replica.cc:330] T 6bbf52826c42434cb40a0238f5d6bbb9 P 752d81f769294f1183734d69013940f2: stopping tablet replica
I20260926 16:43:35.247856 12583 tablet_replica.cc:330] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe: stopping tablet replica
I20260926 16:43:35.248204 12581 raft_consensus.cc:2229] T 6bbf52826c42434cb40a0238f5d6bbb9 P 752d81f769294f1183734d69013940f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:43:35.248512 12581 raft_consensus.cc:2258] T 6bbf52826c42434cb40a0238f5d6bbb9 P 752d81f769294f1183734d69013940f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:43:35.248473 12583 raft_consensus.cc:2229] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:43:35.249075 12583 raft_consensus.cc:2258] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:43:35.249963 12582 tablet_replica.cc:330] T 6bbf52826c42434cb40a0238f5d6bbb9 P 7ce8ee95d02a4b278bb5a80c9e072be7: stopping tablet replica
I20260926 16:43:35.250331 12581 ts_tablet_manager.cc:1905] T 6bbf52826c42434cb40a0238f5d6bbb9 P 752d81f769294f1183734d69013940f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:43:35.250303 12582 raft_consensus.cc:2229] T 6bbf52826c42434cb40a0238f5d6bbb9 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:43:35.250633 12582 raft_consensus.cc:2258] T 6bbf52826c42434cb40a0238f5d6bbb9 P 7ce8ee95d02a4b278bb5a80c9e072be7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:43:35.252614 12583 ts_tablet_manager.cc:1905] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:43:35.252689 12582 ts_tablet_manager.cc:1905] T 6bbf52826c42434cb40a0238f5d6bbb9 P 7ce8ee95d02a4b278bb5a80c9e072be7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:43:35.258006 12581 ts_tablet_manager.cc:1918] T 6bbf52826c42434cb40a0238f5d6bbb9 P 752d81f769294f1183734d69013940f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 16:43:35.258201 12581 log.cc:1196] T 6bbf52826c42434cb40a0238f5d6bbb9 P 752d81f769294f1183734d69013940f2: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-0/wal/wals/6bbf52826c42434cb40a0238f5d6bbb9
I20260926 16:43:35.258800 12581 ts_tablet_manager.cc:1939] T 6bbf52826c42434cb40a0238f5d6bbb9 P 752d81f769294f1183734d69013940f2: Deleting consensus metadata
I20260926 16:43:35.260206 12582 ts_tablet_manager.cc:1918] T 6bbf52826c42434cb40a0238f5d6bbb9 P 7ce8ee95d02a4b278bb5a80c9e072be7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 16:43:35.260404 12582 log.cc:1196] T 6bbf52826c42434cb40a0238f5d6bbb9 P 7ce8ee95d02a4b278bb5a80c9e072be7: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-2/wal/wals/6bbf52826c42434cb40a0238f5d6bbb9
I20260926 16:43:35.260334 11906 catalog_manager.cc:4841] TS 752d81f769294f1183734d69013940f2 (127.9.238.193:34407): tablet 6bbf52826c42434cb40a0238f5d6bbb9 (table test-table [id=6c2e8df8ed0540a38d3c2e17ea5e1e2d]) successfully deleted
I20260926 16:43:35.260957 12582 ts_tablet_manager.cc:1939] T 6bbf52826c42434cb40a0238f5d6bbb9 P 7ce8ee95d02a4b278bb5a80c9e072be7: Deleting consensus metadata
I20260926 16:43:35.261741 12583 ts_tablet_manager.cc:1918] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 16:43:35.261937 12583 log.cc:1196] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790440988577176-10171-0/minicluster-data/ts-1/wal/wals/6bbf52826c42434cb40a0238f5d6bbb9
I20260926 16:43:35.262023 11906 catalog_manager.cc:4841] TS 7ce8ee95d02a4b278bb5a80c9e072be7 (127.9.238.195:44787): tablet 6bbf52826c42434cb40a0238f5d6bbb9 (table test-table [id=6c2e8df8ed0540a38d3c2e17ea5e1e2d]) successfully deleted
I20260926 16:43:35.262445 12583 ts_tablet_manager.cc:1939] T 6bbf52826c42434cb40a0238f5d6bbb9 P d0139f9f3e4e4235ac89980b174840fe: Deleting consensus metadata
I20260926 16:43:35.263736 11907 catalog_manager.cc:4841] TS d0139f9f3e4e4235ac89980b174840fe (127.9.238.194:36687): tablet 6bbf52826c42434cb40a0238f5d6bbb9 (table test-table [id=6c2e8df8ed0540a38d3c2e17ea5e1e2d]) successfully deleted
Sep 26 16:43:35 dist-test-slave-wt2j krb5kdc[11876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441012, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.238.193@KRBTEST.COM
I20260926 16:43:35.278698 12585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:35.238893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49900 (local address 127.9.238.193:34407)
0926 16:43:35.251454 (+ 12561us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:35.251467 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:35.251505 (+ 38us) server_negotiation.cc:407] Connection header received
0926 16:43:35.251627 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:35.251634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:35.251762 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:35.251960 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:35.254129 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:35.255398 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:35.257548 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:35.258498 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:35.260660 (+ 2162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:35.271509 (+ 10849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:35.271532 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:35.271617 (+ 85us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:35.271682 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:35.274966 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:35.275967 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:35.275980 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:35.275987 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:35.276151 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:35.276586 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:35.276595 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:35.276597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:35.277084 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:35.277464 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:35.277793 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 16:43:35.278088 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12369,"thread_start_us":13204,"threads_started":1}
W20260926 16:43:35.279686 12123 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49900
W20260926 16:43:35.280642 10171 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 11992: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:43:35.280903 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 11992
I20260926 16:43:35.312371 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 12132
Sep 26 16:43:35 dist-test-slave-wt2j krb5kdc[11876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441012, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.238.194@KRBTEST.COM
I20260926 16:43:35.331643 12586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:35.313583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57612 (local address 127.9.238.194:36687)
0926 16:43:35.314248 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:35.314253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:35.314271 (+ 18us) server_negotiation.cc:407] Connection header received
0926 16:43:35.314334 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:35.314337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:35.314413 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:35.314531 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:35.316481 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:35.317319 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:35.319088 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:35.319519 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:35.319755 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:35.325019 (+ 5264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:35.325045 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:35.325065 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:35.325114 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:35.328380 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:35.329363 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:35.329370 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:35.329373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:35.329510 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:35.329912 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:35.329915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:35.329917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:35.330332 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:35.330566 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:35.331087 (+ 521us) server_negotiation.cc:299] Negotiation successful
0926 16:43:35.331274 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":327,"threads_started":1}
W20260926 16:43:35.332571 12263 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57612
W20260926 16:43:35.333469 10171 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 12132: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:43:35.333685 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 12132
I20260926 16:43:35.361827 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 12272
Sep 26 16:43:35 dist-test-slave-wt2j krb5kdc[11876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441012, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.9.238.195@KRBTEST.COM
I20260926 16:43:35.381251 12587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:35.362801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53414 (local address 127.9.238.195:44787)
0926 16:43:35.363510 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:35.363515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:35.363536 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:43:35.363674 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:35.363679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:35.363754 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:35.363874 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:35.365853 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:35.366758 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:35.368694 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:35.369209 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:35.369348 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:35.374502 (+ 5154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:35.374520 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:35.374537 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:35.374613 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:35.377806 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:35.378702 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:35.378709 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:35.378712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:35.378849 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:35.379411 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:35.379415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:35.379417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:35.379938 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:35.380240 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:35.380687 (+ 447us) server_negotiation.cc:299] Negotiation successful
0926 16:43:35.380875 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":428,"threads_started":1}
W20260926 16:43:35.382037 12403 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53414
W20260926 16:43:35.382752 10171 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 12272: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:43:35.382957 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 12272
I20260926 16:43:35.408296 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 11889
I20260926 16:43:35.409828 11961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:43:35.551345 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 11889
2026-09-26T16:43:35Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (12421 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:43:35 dist-test-slave-wt2j krb5kdc[12591](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:43:35 dist-test-slave-wt2j krb5kdc[12591](info): set up 2 sockets
Sep 26 16:43:35 dist-test-slave-wt2j krb5kdc[12591](info): commencing operation
krb5kdc: starting...
W20260926 16:43:36.725335 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:43:36 dist-test-slave-wt2j krb5kdc[12591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:43:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:43:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:43:36.960472 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:35199
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:37717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:35199
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:37.306156 12604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:37.306528 12604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:37.306697 12604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:37.314270 12604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:43:37.314445 12604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:37.314569 12604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:37.314669 12604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:43:37.314759 12604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:43:37.325234 12604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:37717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:35199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:35199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:37.326660 12604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:37.328079 12604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:37.345602 12610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:37.345973 12604 server_base.cc:1034] running on GCE node
W20260926 16:43:37.345615 12609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:37.345676 12612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:37.347419 12604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:37.349301 12604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:37.350668 12604 hybrid_clock.cc:648] HybridClock initialized: now 1790441017350613 us; error 52 us; skew 500 ppm
I20260926 16:43:37.351125 12604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:37 dist-test-slave-wt2j krb5kdc[12591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441017, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:37.354861 12604 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:43:37.356782 12604 webserver.cc:460] Webserver started at http://127.9.238.254:42457/ using document root <none> and password file <none>
I20260926 16:43:37.357537 12604 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:37.357641 12604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:37.357959 12604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:37.361186 12604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "342e1aa915674ce48af0cf91f4022841"
format_stamp: "Formatted at 2026-09-26 16:43:37 on dist-test-slave-wt2j"
server_key: "61647d48ac546f24932d4e627af9afb0"
server_key_iv: "2406f5c7ad051ecb09989c9110496cde"
server_key_version: "encryptionkey@0"
I20260926 16:43:37.361967 12604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "342e1aa915674ce48af0cf91f4022841"
format_stamp: "Formatted at 2026-09-26 16:43:37 on dist-test-slave-wt2j"
server_key: "61647d48ac546f24932d4e627af9afb0"
server_key_iv: "2406f5c7ad051ecb09989c9110496cde"
server_key_version: "encryptionkey@0"
I20260926 16:43:37.367645 12604 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260926 16:43:37.372133 12619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:37.374045 12604 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260926 16:43:37.374298 12604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "342e1aa915674ce48af0cf91f4022841"
format_stamp: "Formatted at 2026-09-26 16:43:37 on dist-test-slave-wt2j"
server_key: "61647d48ac546f24932d4e627af9afb0"
server_key_iv: "2406f5c7ad051ecb09989c9110496cde"
server_key_version: "encryptionkey@0"
I20260926 16:43:37.374555 12604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:37.397684 12604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:37.402390 12604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:37.402719 12604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:37.425686 12604 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:35199
I20260926 16:43:37.425802 12670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:35199 every 8 connection(s)
I20260926 16:43:37.427282 12604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:43:37.432338 12671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:37.436579 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 12604
I20260926 16:43:37.436831 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:43:37.437566 10171 external_mini_cluster.cc:1428] Setting key 4b4e5762867e450eb907644850d3859a
I20260926 16:43:37.448524 12671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 342e1aa915674ce48af0cf91f4022841: Bootstrap starting.
I20260926 16:43:37.453183 12671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 342e1aa915674ce48af0cf91f4022841: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:37.454500 12671 log.cc:824] T 00000000000000000000000000000000 P 342e1aa915674ce48af0cf91f4022841: Log is configured to *not* fsync() on all Append() calls
Sep 26 16:43:37 dist-test-slave-wt2j krb5kdc[12591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:37.459584 12671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 342e1aa915674ce48af0cf91f4022841: No bootstrap required, opened a new log
I20260926 16:43:37.468051 12674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:37.440084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48102 (local address 127.9.238.254:35199)
0926 16:43:37.441127 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:37.441146 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:37.441210 (+ 64us) server_negotiation.cc:407] Connection header received
0926 16:43:37.443413 (+ 2203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:37.443453 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:37.444246 (+ 793us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:37.444862 (+ 616us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:37.446300 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:37.448453 (+ 2153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:37.450036 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:37.451382 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:37.451657 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:37.456477 (+ 4820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:37.456495 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:37.456529 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:37.456582 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:37.461573 (+ 4991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:37.462465 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:37.462479 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:37.462497 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:37.462773 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:37.463317 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:37.463328 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:37.463332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:37.464349 (+ 1017us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:37.464740 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:37.465235 (+ 495us) server_negotiation.cc:299] Negotiation successful
0926 16:43:37.465882 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":418,"threads_started":1}
I20260926 16:43:37.472981 12671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 342e1aa915674ce48af0cf91f4022841 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342e1aa915674ce48af0cf91f4022841" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35199 } }
I20260926 16:43:37.473471 12671 raft_consensus.cc:374] T 00000000000000000000000000000000 P 342e1aa915674ce48af0cf91f4022841 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:37.473559 12671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 342e1aa915674ce48af0cf91f4022841 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 342e1aa915674ce48af0cf91f4022841, State: Initialized, Role: FOLLOWER
I20260926 16:43:37.474337 12671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 342e1aa915674ce48af0cf91f4022841 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342e1aa915674ce48af0cf91f4022841" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35199 } }
I20260926 16:43:37.474579 12671 raft_consensus.cc:388] T 00000000000000000000000000000000 P 342e1aa915674ce48af0cf91f4022841 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:43:37.474692 12671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 342e1aa915674ce48af0cf91f4022841 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:43:37.474848 12671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 342e1aa915674ce48af0cf91f4022841 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:37.477789 12671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 342e1aa915674ce48af0cf91f4022841 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342e1aa915674ce48af0cf91f4022841" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35199 } }
I20260926 16:43:37.478628 12671 leader_election.cc:302] T 00000000000000000000000000000000 P 342e1aa915674ce48af0cf91f4022841 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 342e1aa915674ce48af0cf91f4022841; no voters:
I20260926 16:43:37.479592 12671 leader_election.cc:288] T 00000000000000000000000000000000 P 342e1aa915674ce48af0cf91f4022841 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:43:37.479809 12676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 342e1aa915674ce48af0cf91f4022841 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:37.482537 12676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 342e1aa915674ce48af0cf91f4022841 [term 1 LEADER]: Becoming Leader. State: Replica: 342e1aa915674ce48af0cf91f4022841, State: Running, Role: LEADER
I20260926 16:43:37.482657 12671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 342e1aa915674ce48af0cf91f4022841 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:43:37.483297 12676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 342e1aa915674ce48af0cf91f4022841 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342e1aa915674ce48af0cf91f4022841" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35199 } }
I20260926 16:43:37.487246 12678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342e1aa915674ce48af0cf91f4022841 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 342e1aa915674ce48af0cf91f4022841. Latest consensus state: current_term: 1 leader_uuid: "342e1aa915674ce48af0cf91f4022841" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342e1aa915674ce48af0cf91f4022841" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35199 } } }
I20260926 16:43:37.487713 12678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342e1aa915674ce48af0cf91f4022841 [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:37.487367 12677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 342e1aa915674ce48af0cf91f4022841 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "342e1aa915674ce48af0cf91f4022841" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "342e1aa915674ce48af0cf91f4022841" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35199 } } }
I20260926 16:43:37.488047 12677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 342e1aa915674ce48af0cf91f4022841 [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:37.489889 12683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:43:37.496353 12683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:43:37.507987 12683 catalog_manager.cc:1347] Generated new cluster ID: 1afce422e64a47139dca60610240a5bc
I20260926 16:43:37.508179 12683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:43:37.527595 12683 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:43:37.529421 12683 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:43:37.541145 12683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 342e1aa915674ce48af0cf91f4022841: Generated new TSK 0
I20260926 16:43:37.542384 12683 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
I20260926 16:43:37.604249 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:0
--local_ip_for_outbound_sockets=127.9.238.193
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:35199
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:37717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:38.027117 12697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:38.027593 12697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:38.027801 12697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:38.036026 12697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:38.036208 12697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:38.036422 12697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:43:38.048094 12697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:37717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:35199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:38.049883 12697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:38.051599 12697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:38.068068 12702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:38.068100 12703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:38.068235 12705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:38.069643 12697 server_base.cc:1034] running on GCE node
I20260926 16:43:38.070485 12697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:38.071857 12697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:38.073108 12697 hybrid_clock.cc:648] HybridClock initialized: now 1790441018073024 us; error 72 us; skew 500 ppm
I20260926 16:43:38.073506 12697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:38 dist-test-slave-wt2j krb5kdc[12591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441018, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:38.077523 12697 init.cc:377] Logged in from keytab as kudu/127.9.238.193@KRBTEST.COM (short username kudu)
I20260926 16:43:38.079749 12697 webserver.cc:460] Webserver started at http://127.9.238.193:40441/ using document root <none> and password file <none>
I20260926 16:43:38.080641 12697 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:38.080798 12697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:38.081167 12697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:38.084928 12697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/data/instance:
uuid: "0cbaf8bc5d8548628a358a199f5e1998"
format_stamp: "Formatted at 2026-09-26 16:43:38 on dist-test-slave-wt2j"
server_key: "9bcae3bb0ccca78b28a1cd5ca8f00111"
server_key_iv: "5faa0aeb1e4d63fe1df2fd3eb5c991b5"
server_key_version: "encryptionkey@0"
I20260926 16:43:38.085839 12697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance:
uuid: "0cbaf8bc5d8548628a358a199f5e1998"
format_stamp: "Formatted at 2026-09-26 16:43:38 on dist-test-slave-wt2j"
server_key: "9bcae3bb0ccca78b28a1cd5ca8f00111"
server_key_iv: "5faa0aeb1e4d63fe1df2fd3eb5c991b5"
server_key_version: "encryptionkey@0"
I20260926 16:43:38.092666 12697 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 16:43:38.097671 12712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:38.100107 12697 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:43:38.100359 12697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "0cbaf8bc5d8548628a358a199f5e1998"
format_stamp: "Formatted at 2026-09-26 16:43:38 on dist-test-slave-wt2j"
server_key: "9bcae3bb0ccca78b28a1cd5ca8f00111"
server_key_iv: "5faa0aeb1e4d63fe1df2fd3eb5c991b5"
server_key_version: "encryptionkey@0"
I20260926 16:43:38.100651 12697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:38.121277 12697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:38.126174 12697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:38.126530 12697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:38.127856 12697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:38.130085 12697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:38.130198 12697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:38.130360 12697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:38.130437 12697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:38.171056 12697 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:38785
I20260926 16:43:38.171116 12824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:38785 every 8 connection(s)
I20260926 16:43:38.172856 12697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:43:38.182637 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 12697
I20260926 16:43:38.182941 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance
I20260926 16:43:38.183838 10171 external_mini_cluster.cc:1428] Setting key b1e0c99126e68da1028be77682da2b3b
Sep 26 16:43:38 dist-test-slave-wt2j krb5kdc[12591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441018, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:38.200047 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:38.176200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:45349 (local address 127.9.238.254:35199)
0926 16:43:38.176880 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:38.176894 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:38.178587 (+ 1693us) server_negotiation.cc:407] Connection header received
0926 16:43:38.180927 (+ 2340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:38.180932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:38.181024 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:38.181205 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:38.183805 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:38.184787 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:38.187072 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:38.187501 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:38.187700 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:38.193194 (+ 5494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:38.193201 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:38.193205 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:38.193245 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:38.196354 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:38.197294 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:38.197303 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:38.197312 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:38.197436 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:38.197933 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:38.197946 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:38.197948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:38.198254 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:38.198429 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:38.199343 (+ 914us) server_negotiation.cc:299] Negotiation successful
0926 16:43:38.199645 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":298,"threads_started":1}
I20260926 16:43:38.201366 12828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:38.177227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35199 (local address 127.9.238.193:45349)
0926 16:43:38.178336 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:38.178394 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:38.180583 (+ 2189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:38.181520 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:38.181530 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:38.182357 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:38.183431 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:38.183459 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:38.184982 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:38.184988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:38.186855 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:38.186872 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:38.187843 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:38.187853 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:38.188260 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:38.189199 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:38.189239 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:38.192860 (+ 3621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:38.196555 (+ 3695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:38.196568 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:38.196590 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:38.197138 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:38.197571 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:38.197577 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:38.197581 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:38.197801 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:38.198405 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:38.198415 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:38.198851 (+ 436us) client_negotiation.cc:769] Sending connection context
0926 16:43:38.199191 (+ 340us) client_negotiation.cc:240] Negotiation successful
0926 16:43:38.199848 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"thread_start_us":400,"threads_started":1}
I20260926 16:43:38.203608 12825 heartbeater.cc:347] Connected to a master server at 127.9.238.254:35199
I20260926 16:43:38.204129 12825 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:38.205439 12825 heartbeater.cc:511] Master 127.9.238.254:35199 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.238.194@KRBTEST.COM; defaulting to no policy
I20260926 16:43:38.208588 12636 ts_manager.cc:194] Registered new tserver with Master: 0cbaf8bc5d8548628a358a199f5e1998 (127.9.238.193:38785)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.194@KRBTEST.COM" created.
I20260926 16:43:38.210654 12636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.193@KRBTEST.COM'} at 127.9.238.193:45349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
I20260926 16:43:38.240274 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:0
--local_ip_for_outbound_sockets=127.9.238.194
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:35199
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:37717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:38.624547 12831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:38.624854 12831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:38.625005 12831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:38.632401 12831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:38.632530 12831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:38.632717 12831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:43:38.642777 12831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:37717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:35199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:38.644315 12831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:38.645701 12831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:38.665628 12839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:38.665652 12837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:38.666504 12836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:38.667913 12831 server_base.cc:1034] running on GCE node
I20260926 16:43:38.668684 12831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:38.669787 12831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:38.671092 12831 hybrid_clock.cc:648] HybridClock initialized: now 1790441018670996 us; error 66 us; skew 500 ppm
I20260926 16:43:38.671541 12831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:38 dist-test-slave-wt2j krb5kdc[12591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441018, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:38.676017 12831 init.cc:377] Logged in from keytab as kudu/127.9.238.194@KRBTEST.COM (short username kudu)
I20260926 16:43:38.678141 12831 webserver.cc:460] Webserver started at http://127.9.238.194:40871/ using document root <none> and password file <none>
I20260926 16:43:38.679111 12831 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:38.679275 12831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:38.679744 12831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:38.682978 12831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/data/instance:
uuid: "cd58404197a4461ab3bf0c95ad41896a"
format_stamp: "Formatted at 2026-09-26 16:43:38 on dist-test-slave-wt2j"
server_key: "4f8be0762b0c976fe7b24a1cacaa8cce"
server_key_iv: "cc8646daf313a819c942baa663e888ab"
server_key_version: "encryptionkey@0"
I20260926 16:43:38.683951 12831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance:
uuid: "cd58404197a4461ab3bf0c95ad41896a"
format_stamp: "Formatted at 2026-09-26 16:43:38 on dist-test-slave-wt2j"
server_key: "4f8be0762b0c976fe7b24a1cacaa8cce"
server_key_iv: "cc8646daf313a819c942baa663e888ab"
server_key_version: "encryptionkey@0"
I20260926 16:43:38.690405 12831 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 16:43:38.695360 12846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:38.697405 12831 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260926 16:43:38.697655 12831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "cd58404197a4461ab3bf0c95ad41896a"
format_stamp: "Formatted at 2026-09-26 16:43:38 on dist-test-slave-wt2j"
server_key: "4f8be0762b0c976fe7b24a1cacaa8cce"
server_key_iv: "cc8646daf313a819c942baa663e888ab"
server_key_version: "encryptionkey@0"
I20260926 16:43:38.697906 12831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:38.720081 12831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:38.725023 12831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:38.725343 12831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:38.726495 12831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:38.728106 12831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:38.728221 12831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:38.728379 12831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:38.728451 12831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:38.767706 12831 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:39693
I20260926 16:43:38.767742 12958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:39693 every 8 connection(s)
I20260926 16:43:38.769088 12831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:43:38.777278 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 12831
I20260926 16:43:38.777518 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance
I20260926 16:43:38.778110 10171 external_mini_cluster.cc:1428] Setting key 65a1ca5c0126bd45cd9860368680a6e4
Sep 26 16:43:38 dist-test-slave-wt2j krb5kdc[12591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441018, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:38.796401 12961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:38.772406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:46671 (local address 127.9.238.254:35199)
0926 16:43:38.772966 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:38.772974 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:38.774378 (+ 1404us) server_negotiation.cc:407] Connection header received
0926 16:43:38.776507 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:38.776513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:38.776626 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:38.776778 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:38.779170 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:38.780504 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:38.783246 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:38.783897 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:38.784030 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:38.789012 (+ 4982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:38.789018 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:38.789021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:38.789061 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:38.792301 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:38.793497 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:38.793502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:38.793505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:38.793653 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:38.794259 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:38.794265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:38.794268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:38.794607 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:38.794771 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:38.795876 (+ 1105us) server_negotiation.cc:299] Negotiation successful
0926 16:43:38.796076 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":242,"threads_started":1}
I20260926 16:43:38.798130 12962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:38.773252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35199 (local address 127.9.238.194:46671)
0926 16:43:38.774160 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:38.774210 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:38.776019 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:38.777181 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:38.777194 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:38.778060 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:38.778900 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:38.778922 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:38.780716 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:38.780721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:38.782974 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:38.782996 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:38.784174 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:38.784185 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:38.784543 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:38.785385 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:38.785425 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:38.788669 (+ 3244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:38.792500 (+ 3831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:38.792515 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:38.792539 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:38.793322 (+ 783us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:38.793826 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:38.793834 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:38.793842 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:38.794122 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:38.794776 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:38.794787 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:38.795356 (+ 569us) client_negotiation.cc:769] Sending connection context
0926 16:43:38.795766 (+ 410us) client_negotiation.cc:240] Negotiation successful
0926 16:43:38.796430 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":340,"threads_started":1}
WARNING: no policy specified for kudu/127.9.238.195@KRBTEST.COM; defaulting to no policy
I20260926 16:43:38.800602 12959 heartbeater.cc:347] Connected to a master server at 127.9.238.254:35199
I20260926 16:43:38.801041 12959 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:38.802191 12959 heartbeater.cc:511] Master 127.9.238.254:35199 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.195@KRBTEST.COM" created.
I20260926 16:43:38.804284 12636 ts_manager.cc:194] Registered new tserver with Master: cd58404197a4461ab3bf0c95ad41896a (127.9.238.194:39693)
I20260926 16:43:38.805418 12636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.194@KRBTEST.COM'} at 127.9.238.194:46671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
I20260926 16:43:38.828202 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:0
--local_ip_for_outbound_sockets=127.9.238.195
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:35199
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:37717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:39.189690 12965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:39.189996 12965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:39.190104 12965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:39.198345 12965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:39.198487 12965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:39.198619 12965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:43:39.209789 12965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:37717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:35199
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:39.211124 12965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:39.212452 12965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 16:43:39.214890 12825 heartbeater.cc:503] Master 127.9.238.254:35199 was elected leader, sending a full tablet report...
W20260926 16:43:39.228242 12971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:39.228281 12973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:39.228556 12970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:39.230018 12965 server_base.cc:1034] running on GCE node
I20260926 16:43:39.230825 12965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:39.232019 12965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:39.233345 12965 hybrid_clock.cc:648] HybridClock initialized: now 1790441019233275 us; error 60 us; skew 500 ppm
I20260926 16:43:39.233747 12965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:39 dist-test-slave-wt2j krb5kdc[12591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441019, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:39.237587 12965 init.cc:377] Logged in from keytab as kudu/127.9.238.195@KRBTEST.COM (short username kudu)
I20260926 16:43:39.239499 12965 webserver.cc:460] Webserver started at http://127.9.238.195:46683/ using document root <none> and password file <none>
I20260926 16:43:39.240366 12965 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:39.240536 12965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:39.240880 12965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:39.244318 12965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/data/instance:
uuid: "7a2d11fa812d41ee948be3a888e435ff"
format_stamp: "Formatted at 2026-09-26 16:43:39 on dist-test-slave-wt2j"
server_key: "10d40105e3b11531ab7934c2293f0531"
server_key_iv: "ce1bb09b56698d09c3b67575fab2fbe2"
server_key_version: "encryptionkey@0"
I20260926 16:43:39.245255 12965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance:
uuid: "7a2d11fa812d41ee948be3a888e435ff"
format_stamp: "Formatted at 2026-09-26 16:43:39 on dist-test-slave-wt2j"
server_key: "10d40105e3b11531ab7934c2293f0531"
server_key_iv: "ce1bb09b56698d09c3b67575fab2fbe2"
server_key_version: "encryptionkey@0"
I20260926 16:43:39.251896 12965 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 16:43:39.256302 12980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:39.258263 12965 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 16:43:39.258502 12965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "7a2d11fa812d41ee948be3a888e435ff"
format_stamp: "Formatted at 2026-09-26 16:43:39 on dist-test-slave-wt2j"
server_key: "10d40105e3b11531ab7934c2293f0531"
server_key_iv: "ce1bb09b56698d09c3b67575fab2fbe2"
server_key_version: "encryptionkey@0"
I20260926 16:43:39.258773 12965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:39.282258 12965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:39.287007 12965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:39.287277 12965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:39.288381 12965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:39.289958 12965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:39.290045 12965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:39.290141 12965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:39.290194 12965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:39.331857 12965 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:44047
I20260926 16:43:39.331890 13092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:44047 every 8 connection(s)
I20260926 16:43:39.333248 12965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:43:39.341115 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 12965
I20260926 16:43:39.341388 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance
I20260926 16:43:39.341894 10171 external_mini_cluster.cc:1428] Setting key 3afe2b2fc99b3f1b81531ee803152f1b
Sep 26 16:43:39 dist-test-slave-wt2j krb5kdc[12591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441019, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:39.357894 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:39.336703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:55285 (local address 127.9.238.254:35199)
0926 16:43:39.337287 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:39.337292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:39.338410 (+ 1118us) server_negotiation.cc:407] Connection header received
0926 16:43:39.340658 (+ 2248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:39.340664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:39.340794 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:39.340929 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:39.342905 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:39.344101 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:39.346081 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:39.346579 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:39.346699 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:39.350833 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:39.350840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:39.350845 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:39.350891 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:39.354028 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:39.354961 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:39.354966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:39.354968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:39.355117 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:39.355634 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:39.355638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:39.355640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:39.356004 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:39.356141 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:39.357145 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0926 16:43:39.357473 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":288,"threads_started":1}
I20260926 16:43:39.359020 13096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:39.337262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35199 (local address 127.9.238.195:55285)
0926 16:43:39.338189 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:39.338248 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:39.340320 (+ 2072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:39.341219 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:39.341229 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:39.341941 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:39.342650 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:39.342670 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:39.344345 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:39.344351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:39.345932 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:39.345945 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:39.346840 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:39.346846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:39.347092 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:39.347912 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:39.347942 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:39.350550 (+ 2608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:39.354256 (+ 3706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:39.354264 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:39.354280 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:39.354795 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:39.355244 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:39.355247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:39.355250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:39.355452 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:39.356156 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:39.356164 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:39.356606 (+ 442us) client_negotiation.cc:769] Sending connection context
0926 16:43:39.356980 (+ 374us) client_negotiation.cc:240] Negotiation successful
0926 16:43:39.357554 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":360,"threads_started":1}
I20260926 16:43:39.360669 13093 heartbeater.cc:347] Connected to a master server at 127.9.238.254:35199
I20260926 16:43:39.361129 13093 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:39.362174 13093 heartbeater.cc:511] Master 127.9.238.254:35199 requested a full tablet report, sending...
I20260926 16:43:39.364188 12636 ts_manager.cc:194] Registered new tserver with Master: 7a2d11fa812d41ee948be3a888e435ff (127.9.238.195:44047)
I20260926 16:43:39.365266 12636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.195@KRBTEST.COM'} at 127.9.238.195:55285
I20260926 16:43:39.371701 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 16:43:39.411868 13095 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:43:39.408798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48108 (local address 127.9.238.254:35199)
0926 16:43:39.409204 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:39.409212 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:39.409504 (+ 292us) server_negotiation.cc:407] Connection header received
0926 16:43:39.409699 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:39.409704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:39.409784 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:39.409895 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:39.411548 (+ 1653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48108: BlockingRecv error: recv got EOF from 127.0.0.1:48108 (error 108)
Metrics: {"server-negotiator.queue_time_us":151}
I20260926 16:43:39.416317 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 12697
W20260926 16:43:39.420284 10171 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 12697: Not authorized: Client connection negotiation failed: client connection to 127.9.238.193:38785: server requires authentication, but client does not have Kerberos credentials available
I20260926 16:43:39.420550 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 12697
I20260926 16:43:39.438369 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 12831
W20260926 16:43:39.441898 10171 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 12831: Not authorized: Client connection negotiation failed: client connection to 127.9.238.194:39693: server requires authentication, but client does not have Kerberos credentials available
I20260926 16:43:39.442150 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 12831
W20260926 16:43:39.442133 13106 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:43:39.439797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.9.238.194:39693)
0926 16:43:39.440465 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:39.440473 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:39.440503 (+ 30us) server_negotiation.cc:407] Connection header received
0926 16:43:39.440571 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:39.440578 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:39.440807 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:39.441005 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:39.441780 (+ 775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50320: BlockingRecv error: recv got EOF from 127.0.0.1:50320 (error 108)
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":319,"threads_started":1}
I20260926 16:43:39.462283 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 12965
W20260926 16:43:39.466223 10171 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 12965: Not authorized: Client connection negotiation failed: client connection to 127.9.238.195:44047: server requires authentication, but client does not have Kerberos credentials available
I20260926 16:43:39.466375 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 12965
I20260926 16:43:39.481789 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 12604
I20260926 16:43:39.483490 12666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:43:39.594345 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 12604
2026-09-26T16:43:39Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4042 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:43:39 dist-test-slave-wt2j krb5kdc[13111](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:43:39 dist-test-slave-wt2j krb5kdc[13111](info): set up 2 sockets
Sep 26 16:43:39 dist-test-slave-wt2j krb5kdc[13111](info): commencing operation
krb5kdc: starting...
W20260926 16:43:40.761673 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:43:40 dist-test-slave-wt2j krb5kdc[13111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:43:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:43:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:43:40.992606 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:32967
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:38901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:32967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:41.403455 13124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:41.403936 13124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:41.404162 13124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:41.412573 13124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:43:41.412752 13124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:41.412871 13124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:41.413008 13124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:43:41.413112 13124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:43:41.424690 13124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:38901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:32967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:32967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:41.426652 13124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:41.428467 13124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:41.442832 13130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:41.443006 13132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:41.443300 13129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:41.444736 13124 server_base.cc:1034] running on GCE node
I20260926 16:43:41.445750 13124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:41.447379 13124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:41.448613 13124 hybrid_clock.cc:648] HybridClock initialized: now 1790441021448550 us; error 51 us; skew 500 ppm
I20260926 16:43:41.448992 13124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:41 dist-test-slave-wt2j krb5kdc[13111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441021, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:41.465736 13124 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:43:41.467870 13124 webserver.cc:460] Webserver started at http://127.9.238.254:42683/ using document root <none> and password file <none>
I20260926 16:43:41.468713 13124 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:41.468858 13124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:41.469223 13124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:41.472573 13124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "73404674727b420ca48004f5f5cf7d06"
format_stamp: "Formatted at 2026-09-26 16:43:41 on dist-test-slave-wt2j"
server_key: "0269edcf6f426a1e93cd6c5ef62ae0f0"
server_key_iv: "bb168648e4cfe1431c64158117ad51d1"
server_key_version: "encryptionkey@0"
I20260926 16:43:41.473814 13124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "73404674727b420ca48004f5f5cf7d06"
format_stamp: "Formatted at 2026-09-26 16:43:41 on dist-test-slave-wt2j"
server_key: "0269edcf6f426a1e93cd6c5ef62ae0f0"
server_key_iv: "bb168648e4cfe1431c64158117ad51d1"
server_key_version: "encryptionkey@0"
I20260926 16:43:41.480952 13124 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.007s
I20260926 16:43:41.485827 13139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:41.488054 13124 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 16:43:41.488313 13124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "73404674727b420ca48004f5f5cf7d06"
format_stamp: "Formatted at 2026-09-26 16:43:41 on dist-test-slave-wt2j"
server_key: "0269edcf6f426a1e93cd6c5ef62ae0f0"
server_key_iv: "bb168648e4cfe1431c64158117ad51d1"
server_key_version: "encryptionkey@0"
I20260926 16:43:41.488658 13124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:41.508535 13124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:41.514261 13124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:41.514631 13124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:41.537855 13124 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:32967
I20260926 16:43:41.537896 13190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:32967 every 8 connection(s)
I20260926 16:43:41.539710 13124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:43:41.544457 13191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:41.547767 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 13124
I20260926 16:43:41.548043 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:43:41.548848 10171 external_mini_cluster.cc:1428] Setting key 2843c7e545684034b9e74674dc00cada
I20260926 16:43:41.559113 13191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73404674727b420ca48004f5f5cf7d06: Bootstrap starting.
I20260926 16:43:41.562942 13191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73404674727b420ca48004f5f5cf7d06: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:41.564108 13191 log.cc:824] T 00000000000000000000000000000000 P 73404674727b420ca48004f5f5cf7d06: Log is configured to *not* fsync() on all Append() calls
Sep 26 16:43:41 dist-test-slave-wt2j krb5kdc[13111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:41.568164 13191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73404674727b420ca48004f5f5cf7d06: No bootstrap required, opened a new log
I20260926 16:43:41.579288 13194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:41.551531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46574 (local address 127.9.238.254:32967)
0926 16:43:41.552737 (+ 1206us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:41.552756 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:41.552811 (+ 55us) server_negotiation.cc:407] Connection header received
0926 16:43:41.554790 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:41.554824 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:41.555431 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:41.555957 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:41.557760 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:41.559792 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:41.561250 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:41.562330 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:41.562546 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:41.568256 (+ 5710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:41.568270 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:41.568293 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:41.568347 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:41.573278 (+ 4931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:41.574322 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:41.574332 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:41.574343 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:41.574593 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:41.575061 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:41.575067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:41.575071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:41.575985 (+ 914us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:41.576331 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:41.576945 (+ 614us) server_negotiation.cc:299] Negotiation successful
0926 16:43:41.577504 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":848,"thread_start_us":541,"threads_started":1}
I20260926 16:43:41.580417 13191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73404674727b420ca48004f5f5cf7d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73404674727b420ca48004f5f5cf7d06" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 32967 } }
I20260926 16:43:41.580849 13191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73404674727b420ca48004f5f5cf7d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:41.580971 13191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73404674727b420ca48004f5f5cf7d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73404674727b420ca48004f5f5cf7d06, State: Initialized, Role: FOLLOWER
I20260926 16:43:41.581719 13191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73404674727b420ca48004f5f5cf7d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73404674727b420ca48004f5f5cf7d06" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 32967 } }
I20260926 16:43:41.582008 13191 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73404674727b420ca48004f5f5cf7d06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:43:41.582180 13191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73404674727b420ca48004f5f5cf7d06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:43:41.582391 13191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73404674727b420ca48004f5f5cf7d06 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:41.585119 10171 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:43:41.585575 13191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73404674727b420ca48004f5f5cf7d06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73404674727b420ca48004f5f5cf7d06" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 32967 } }
I20260926 16:43:41.586200 13191 leader_election.cc:302] T 00000000000000000000000000000000 P 73404674727b420ca48004f5f5cf7d06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73404674727b420ca48004f5f5cf7d06; no voters:
I20260926 16:43:41.586905 13191 leader_election.cc:288] T 00000000000000000000000000000000 P 73404674727b420ca48004f5f5cf7d06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:43:41.587188 13196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73404674727b420ca48004f5f5cf7d06 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:41.589481 13191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73404674727b420ca48004f5f5cf7d06 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:43:41.597752 13196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73404674727b420ca48004f5f5cf7d06 [term 1 LEADER]: Becoming Leader. State: Replica: 73404674727b420ca48004f5f5cf7d06, State: Running, Role: LEADER
I20260926 16:43:41.598738 13196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73404674727b420ca48004f5f5cf7d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73404674727b420ca48004f5f5cf7d06" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 32967 } }
I20260926 16:43:41.603996 13197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73404674727b420ca48004f5f5cf7d06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73404674727b420ca48004f5f5cf7d06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73404674727b420ca48004f5f5cf7d06" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 32967 } } }
I20260926 16:43:41.604571 13197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73404674727b420ca48004f5f5cf7d06 [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:41.605332 13198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73404674727b420ca48004f5f5cf7d06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73404674727b420ca48004f5f5cf7d06. Latest consensus state: current_term: 1 leader_uuid: "73404674727b420ca48004f5f5cf7d06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73404674727b420ca48004f5f5cf7d06" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 32967 } } }
I20260926 16:43:41.605789 13198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73404674727b420ca48004f5f5cf7d06 [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:41.606992 13211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:43:41.614562 13211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:43:41.626276 13211 catalog_manager.cc:1347] Generated new cluster ID: 41168e85118a4c28b3cbe0c575bca5ff
I20260926 16:43:41.626413 13211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:43:41.640398 13211 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:43:41.642596 13211 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:43:41.646667 13194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:41.634624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46588 (local address 127.9.238.254:32967)
0926 16:43:41.634897 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:41.634917 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:41.636611 (+ 1694us) server_negotiation.cc:407] Connection header received
0926 16:43:41.636984 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:41.636992 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:41.637135 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:41.637238 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:41.638953 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:41.640190 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:41.642282 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:41.642985 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:41.643217 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:41.644681 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:41.644689 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:41.644694 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:43:41.645663 (+ 969us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:41.645758 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:43:41.645801 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:41.645933 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:41.646111 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 16:43:41.646274 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 16:43:41.652665 13211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73404674727b420ca48004f5f5cf7d06: Generated new TSK 0
I20260926 16:43:41.653889 13211 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:43:41.665122 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 13124
I20260926 16:43:41.666817 13186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:43:41.758026 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 13124
2026-09-26T16:43:41Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2153 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:43:41 dist-test-slave-wt2j krb5kdc[13225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:43:41 dist-test-slave-wt2j krb5kdc[13225](info): set up 2 sockets
Sep 26 16:43:41 dist-test-slave-wt2j krb5kdc[13225](info): commencing operation
krb5kdc: starting...
W20260926 16:43:42.885890 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:43:43 dist-test-slave-wt2j krb5kdc[13225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:43:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:43:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:43:43.129324 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:43165
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:43165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:43.489687 13238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:43.490020 13238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:43.490113 13238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:43.497903 13238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:43:43.498040 13238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:43.498091 13238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:43.498142 13238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:43:43.498180 13238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:43:43.508560 13238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:43165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:43165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:43.509963 13238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:43.511269 13238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:43.525161 13244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:43.525691 13243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:43.525746 13246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:43.527053 13238 server_base.cc:1034] running on GCE node
I20260926 16:43:43.528105 13238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:43.529740 13238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:43.531033 13238 hybrid_clock.cc:648] HybridClock initialized: now 1790441023530950 us; error 64 us; skew 500 ppm
I20260926 16:43:43.531530 13238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:43 dist-test-slave-wt2j krb5kdc[13225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441023, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:43.535390 13238 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:43:43.537297 13238 webserver.cc:460] Webserver started at http://127.9.238.254:38957/ using document root <none> and password file <none>
I20260926 16:43:43.538040 13238 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:43.538197 13238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:43.538523 13238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:43.541706 13238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "ee67aca55e974a489a69bb902edc29c4"
format_stamp: "Formatted at 2026-09-26 16:43:43 on dist-test-slave-wt2j"
server_key: "00fa8770757bd5469f98e84c7a68f875"
server_key_iv: "b1a3ed3270b6f906048a3f236a7ec51f"
server_key_version: "encryptionkey@0"
I20260926 16:43:43.542495 13238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "ee67aca55e974a489a69bb902edc29c4"
format_stamp: "Formatted at 2026-09-26 16:43:43 on dist-test-slave-wt2j"
server_key: "00fa8770757bd5469f98e84c7a68f875"
server_key_iv: "b1a3ed3270b6f906048a3f236a7ec51f"
server_key_version: "encryptionkey@0"
I20260926 16:43:43.548700 13238 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 16:43:43.553211 13253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:43.555333 13238 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 16:43:43.555650 13238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "ee67aca55e974a489a69bb902edc29c4"
format_stamp: "Formatted at 2026-09-26 16:43:43 on dist-test-slave-wt2j"
server_key: "00fa8770757bd5469f98e84c7a68f875"
server_key_iv: "b1a3ed3270b6f906048a3f236a7ec51f"
server_key_version: "encryptionkey@0"
I20260926 16:43:43.555917 13238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:43.584102 13238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:43.589394 13238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:43.589736 13238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:43.612239 13238 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:43165
I20260926 16:43:43.612298 13304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:43165 every 8 connection(s)
I20260926 16:43:43.613760 13238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:43:43.615747 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 13238
I20260926 16:43:43.616015 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:43:43.616904 10171 external_mini_cluster.cc:1428] Setting key 2ad0ad5a5f51ff6cb5b2c2665042d25f
I20260926 16:43:43.619992 13305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 16:43:43 dist-test-slave-wt2j krb5kdc[13225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:43.638700 13305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee67aca55e974a489a69bb902edc29c4: Bootstrap starting.
I20260926 16:43:43.647442 13305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee67aca55e974a489a69bb902edc29c4: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:43.649969 13305 log.cc:824] T 00000000000000000000000000000000 P ee67aca55e974a489a69bb902edc29c4: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:43.654028 13308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:43.619902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47636 (local address 127.9.238.254:43165)
0926 16:43:43.620894 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:43.620907 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:43.620964 (+ 57us) server_negotiation.cc:407] Connection header received
0926 16:43:43.622774 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:43.622800 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:43.623259 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:43.623738 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:43.625884 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:43.627454 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:43.629549 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:43.630341 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:43.630506 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:43.637219 (+ 6713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:43.637234 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:43.637276 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:43.637346 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:43.645206 (+ 7860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:43.646797 (+ 1591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:43.646811 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:43.646824 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:43.647164 (+ 340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:43.647861 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:43.647872 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:43.647878 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:43.649255 (+ 1377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:43.649744 (+ 489us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:43.651501 (+ 1757us) server_negotiation.cc:299] Negotiation successful
0926 16:43:43.652173 (+ 672us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749,"thread_start_us":433,"threads_started":1}
I20260926 16:43:43.656046 13305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee67aca55e974a489a69bb902edc29c4: No bootstrap required, opened a new log
I20260926 16:43:43.671255 13305 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee67aca55e974a489a69bb902edc29c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee67aca55e974a489a69bb902edc29c4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 43165 } }
I20260926 16:43:43.671991 13305 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee67aca55e974a489a69bb902edc29c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:43.672132 13305 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee67aca55e974a489a69bb902edc29c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee67aca55e974a489a69bb902edc29c4, State: Initialized, Role: FOLLOWER
I20260926 16:43:43.673015 13305 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee67aca55e974a489a69bb902edc29c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee67aca55e974a489a69bb902edc29c4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 43165 } }
I20260926 16:43:43.673328 13305 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee67aca55e974a489a69bb902edc29c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:43:43.673504 13305 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee67aca55e974a489a69bb902edc29c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:43:43.673722 13305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee67aca55e974a489a69bb902edc29c4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:43.677173 13305 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee67aca55e974a489a69bb902edc29c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee67aca55e974a489a69bb902edc29c4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 43165 } }
I20260926 16:43:43.678087 13305 leader_election.cc:302] T 00000000000000000000000000000000 P ee67aca55e974a489a69bb902edc29c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee67aca55e974a489a69bb902edc29c4; no voters:
I20260926 16:43:43.679066 13305 leader_election.cc:288] T 00000000000000000000000000000000 P ee67aca55e974a489a69bb902edc29c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:43:43.679374 13310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee67aca55e974a489a69bb902edc29c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:43.680394 13310 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee67aca55e974a489a69bb902edc29c4 [term 1 LEADER]: Becoming Leader. State: Replica: ee67aca55e974a489a69bb902edc29c4, State: Running, Role: LEADER
I20260926 16:43:43.681249 13310 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee67aca55e974a489a69bb902edc29c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee67aca55e974a489a69bb902edc29c4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 43165 } }
I20260926 16:43:43.682317 13305 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee67aca55e974a489a69bb902edc29c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:43:43.685808 13312 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee67aca55e974a489a69bb902edc29c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee67aca55e974a489a69bb902edc29c4. Latest consensus state: current_term: 1 leader_uuid: "ee67aca55e974a489a69bb902edc29c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee67aca55e974a489a69bb902edc29c4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 43165 } } }
I20260926 16:43:43.686213 13312 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee67aca55e974a489a69bb902edc29c4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:43.687618 13311 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee67aca55e974a489a69bb902edc29c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee67aca55e974a489a69bb902edc29c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee67aca55e974a489a69bb902edc29c4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 43165 } } }
I20260926 16:43:43.688061 13311 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee67aca55e974a489a69bb902edc29c4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:43.690485 13317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:43:43.697002 13317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:43:43.706713 13317 catalog_manager.cc:1347] Generated new cluster ID: e0bbe375669d4827b7d93d2d2d93318e
I20260926 16:43:43.706873 13317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:43:43.716804 13317 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:43:43.718802 13317 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:43:43.741436 13317 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee67aca55e974a489a69bb902edc29c4: Generated new TSK 0
I20260926 16:43:43.742530 13317 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
I20260926 16:43:43.804682 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:0
--local_ip_for_outbound_sockets=127.9.238.193
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:43165
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:44.146052 13332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:44.146358 13332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:44.146454 13332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:44.154161 13332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:44.154286 13332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:44.154405 13332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:43:44.164712 13332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:43165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:44.166007 13332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:44.167260 13332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:44.182931 13337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:44.183069 13338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:44.183002 13340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:44.184190 13332 server_base.cc:1034] running on GCE node
I20260926 16:43:44.185107 13332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:44.186426 13332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:44.187683 13332 hybrid_clock.cc:648] HybridClock initialized: now 1790441024187591 us; error 80 us; skew 500 ppm
I20260926 16:43:44.188099 13332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:44 dist-test-slave-wt2j krb5kdc[13225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441024, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:44.192406 13332 init.cc:377] Logged in from keytab as kudu/127.9.238.193@KRBTEST.COM (short username kudu)
I20260926 16:43:44.194386 13332 webserver.cc:460] Webserver started at http://127.9.238.193:36075/ using document root <none> and password file <none>
I20260926 16:43:44.195322 13332 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:44.195500 13332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:44.195922 13332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:44.200148 13332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/data/instance:
uuid: "6b583f156311445d82683ea726472391"
format_stamp: "Formatted at 2026-09-26 16:43:44 on dist-test-slave-wt2j"
server_key: "af43486fe95ae59c5223fc485aadee15"
server_key_iv: "e24c4f2b5b91206ecf85a930e11ec465"
server_key_version: "encryptionkey@0"
I20260926 16:43:44.201191 13332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance:
uuid: "6b583f156311445d82683ea726472391"
format_stamp: "Formatted at 2026-09-26 16:43:44 on dist-test-slave-wt2j"
server_key: "af43486fe95ae59c5223fc485aadee15"
server_key_iv: "e24c4f2b5b91206ecf85a930e11ec465"
server_key_version: "encryptionkey@0"
I20260926 16:43:44.208293 13332 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260926 16:43:44.212803 13347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:44.214701 13332 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:43:44.214946 13332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "6b583f156311445d82683ea726472391"
format_stamp: "Formatted at 2026-09-26 16:43:44 on dist-test-slave-wt2j"
server_key: "af43486fe95ae59c5223fc485aadee15"
server_key_iv: "e24c4f2b5b91206ecf85a930e11ec465"
server_key_version: "encryptionkey@0"
I20260926 16:43:44.215193 13332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:44.240447 13332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:44.246230 13332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:44.246613 13332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:44.247923 13332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:44.249589 13332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:44.249701 13332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:44.249867 13332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:44.249946 13332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:44.290692 13332 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:36013
I20260926 16:43:44.290772 13459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:36013 every 8 connection(s)
I20260926 16:43:44.292357 13332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:43:44.295364 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 13332
I20260926 16:43:44.295593 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance
I20260926 16:43:44.296330 10171 external_mini_cluster.cc:1428] Setting key 85696245c370cfb67809d6627087c43f
Sep 26 16:43:44 dist-test-slave-wt2j krb5kdc[13225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441024, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.238.194@KRBTEST.COM; defaulting to no policy
I20260926 16:43:44.319727 13462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:44.295916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:60843 (local address 127.9.238.254:43165)
0926 16:43:44.296554 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:44.296561 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:44.297993 (+ 1432us) server_negotiation.cc:407] Connection header received
0926 16:43:44.300222 (+ 2229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:44.300230 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:44.300330 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:44.300466 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:44.302317 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:44.303601 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:44.306461 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:44.306882 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:44.306997 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:44.312295 (+ 5298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:44.312300 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:44.312304 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:44.312338 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:44.315021 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:44.316262 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:44.316269 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:44.316272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:44.316556 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:44.317190 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:44.317194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:44.317196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:44.317652 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:44.317801 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:44.318903 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0926 16:43:44.319153 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":304,"threads_started":1}
I20260926 16:43:44.320986 13463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:44.296670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:43165 (local address 127.9.238.193:60843)
0926 16:43:44.297691 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:44.297742 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:44.299861 (+ 2119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:44.300712 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:44.300722 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:44.301410 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:44.302115 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:44.302134 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:44.303840 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:44.303856 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:44.306255 (+ 2399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:44.306278 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:44.307171 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:44.307181 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:44.307587 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:44.308544 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:44.308582 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:44.312030 (+ 3448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:44.315190 (+ 3160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:44.315203 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:44.315228 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:44.316027 (+ 799us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:44.316742 (+ 715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:44.316749 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:44.316755 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:44.317037 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:44.317821 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:44.317832 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:44.318376 (+ 544us) client_negotiation.cc:769] Sending connection context
0926 16:43:44.318742 (+ 366us) client_negotiation.cc:240] Negotiation successful
0926 16:43:44.319458 (+ 716us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":451,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.194@KRBTEST.COM" created.
I20260926 16:43:44.323504 13460 heartbeater.cc:347] Connected to a master server at 127.9.238.254:43165
I20260926 16:43:44.324007 13460 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:44.324954 13460 heartbeater.cc:511] Master 127.9.238.254:43165 requested a full tablet report, sending...
I20260926 16:43:44.327889 13270 ts_manager.cc:194] Registered new tserver with Master: 6b583f156311445d82683ea726472391 (127.9.238.193:36013)
I20260926 16:43:44.329790 13270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.193@KRBTEST.COM'} at 127.9.238.193:60843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
I20260926 16:43:44.352264 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:0
--local_ip_for_outbound_sockets=127.9.238.194
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:43165
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:44.690495 13466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:44.690833 13466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:44.691004 13466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:44.698418 13466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:44.698580 13466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:44.698810 13466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:43:44.708979 13466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:43165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:44.710409 13466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:44.711790 13466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:44.728559 13474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:44.728524 13471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:44.728611 13472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:44.730412 13466 server_base.cc:1034] running on GCE node
I20260926 16:43:44.731181 13466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:44.732409 13466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:44.733696 13466 hybrid_clock.cc:648] HybridClock initialized: now 1790441024733616 us; error 86 us; skew 500 ppm
I20260926 16:43:44.734144 13466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:44 dist-test-slave-wt2j krb5kdc[13225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441024, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:44.737919 13466 init.cc:377] Logged in from keytab as kudu/127.9.238.194@KRBTEST.COM (short username kudu)
I20260926 16:43:44.739878 13466 webserver.cc:460] Webserver started at http://127.9.238.194:34871/ using document root <none> and password file <none>
I20260926 16:43:44.740621 13466 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:44.740773 13466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:44.741107 13466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:44.744382 13466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/data/instance:
uuid: "176e818e5b7c453b856b466aaab6fcf9"
format_stamp: "Formatted at 2026-09-26 16:43:44 on dist-test-slave-wt2j"
server_key: "3a4cc6e5099acd0c573eb8637f654fbd"
server_key_iv: "4b551f873290ff03561abe8e8f230e71"
server_key_version: "encryptionkey@0"
I20260926 16:43:44.745209 13466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance:
uuid: "176e818e5b7c453b856b466aaab6fcf9"
format_stamp: "Formatted at 2026-09-26 16:43:44 on dist-test-slave-wt2j"
server_key: "3a4cc6e5099acd0c573eb8637f654fbd"
server_key_iv: "4b551f873290ff03561abe8e8f230e71"
server_key_version: "encryptionkey@0"
I20260926 16:43:44.751081 13466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260926 16:43:44.755060 13481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:44.756778 13466 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:43:44.756976 13466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "176e818e5b7c453b856b466aaab6fcf9"
format_stamp: "Formatted at 2026-09-26 16:43:44 on dist-test-slave-wt2j"
server_key: "3a4cc6e5099acd0c573eb8637f654fbd"
server_key_iv: "4b551f873290ff03561abe8e8f230e71"
server_key_version: "encryptionkey@0"
I20260926 16:43:44.757180 13466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:44.790905 13466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:44.795750 13466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:44.796052 13466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:44.797161 13466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:44.798734 13466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:44.798839 13466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:44.799018 13466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:44.799098 13466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:44.839048 13466 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:38069
I20260926 16:43:44.839165 13593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:38069 every 8 connection(s)
I20260926 16:43:44.840524 13466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:43:44.847599 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 13466
I20260926 16:43:44.847838 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance
I20260926 16:43:44.848510 10171 external_mini_cluster.cc:1428] Setting key 1066eccf23b0e7267d149249554f6597
Sep 26 16:43:44 dist-test-slave-wt2j krb5kdc[13225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441024, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:44.869444 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:44.843721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:55697 (local address 127.9.238.254:43165)
0926 16:43:44.844383 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:44.844389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:44.845216 (+ 827us) server_negotiation.cc:407] Connection header received
0926 16:43:44.847880 (+ 2664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:44.847905 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:44.848014 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:44.848139 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:44.850535 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:44.851411 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:44.854894 (+ 3483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:44.855403 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:44.855524 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:44.861469 (+ 5945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:44.861478 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:44.861493 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:44.861546 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:44.865429 (+ 3883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:44.866442 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:44.866451 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:44.866457 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:44.866632 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:44.867112 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:44.867120 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:44.867126 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:44.867540 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:44.867745 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:44.868640 (+ 895us) server_negotiation.cc:299] Negotiation successful
0926 16:43:44.868910 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":332,"threads_started":1}
I20260926 16:43:44.870613 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:44.844151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:43165 (local address 127.9.238.194:55697)
0926 16:43:44.845012 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:44.845068 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:44.847412 (+ 2344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:44.848500 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:44.848516 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:44.849414 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:44.850303 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:44.850323 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:44.851625 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:44.851631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:44.854667 (+ 3036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:44.854694 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:44.855738 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:44.855748 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:44.856163 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:44.857212 (+ 1049us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:44.857256 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:44.861084 (+ 3828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:44.865656 (+ 4572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:44.865666 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:44.865685 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:44.866266 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:44.866748 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:44.866751 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:44.866753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:44.866975 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:44.867713 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:44.867721 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:44.868185 (+ 464us) client_negotiation.cc:769] Sending connection context
0926 16:43:44.868480 (+ 295us) client_negotiation.cc:240] Negotiation successful
0926 16:43:44.869006 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":344,"threads_started":1}
I20260926 16:43:44.872476 13594 heartbeater.cc:347] Connected to a master server at 127.9.238.254:43165
I20260926 16:43:44.872852 13594 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.9.238.195@KRBTEST.COM; defaulting to no policy
I20260926 16:43:44.873826 13594 heartbeater.cc:511] Master 127.9.238.254:43165 requested a full tablet report, sending...
I20260926 16:43:44.875931 13270 ts_manager.cc:194] Registered new tserver with Master: 176e818e5b7c453b856b466aaab6fcf9 (127.9.238.194:38069)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.195@KRBTEST.COM" created.
I20260926 16:43:44.877012 13270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.194@KRBTEST.COM'} at 127.9.238.194:55697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
I20260926 16:43:44.904109 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:0
--local_ip_for_outbound_sockets=127.9.238.195
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:43165
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:45.233903 13600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:45.234225 13600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:45.234321 13600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:45.241895 13600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:45.242044 13600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:45.242163 13600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:43:45.252162 13600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:43165
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:45.253531 13600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:45.254881 13600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:45.270998 13606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:45.271006 13608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:45.271260 13605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:45.272266 13600 server_base.cc:1034] running on GCE node
I20260926 16:43:45.273054 13600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:45.274159 13600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:45.275490 13600 hybrid_clock.cc:648] HybridClock initialized: now 1790441025275360 us; error 87 us; skew 500 ppm
I20260926 16:43:45.275905 13600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:45 dist-test-slave-wt2j krb5kdc[13225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441025, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:45.279902 13600 init.cc:377] Logged in from keytab as kudu/127.9.238.195@KRBTEST.COM (short username kudu)
I20260926 16:43:45.281934 13600 webserver.cc:460] Webserver started at http://127.9.238.195:35277/ using document root <none> and password file <none>
I20260926 16:43:45.282719 13600 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:45.282852 13600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:45.283181 13600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:45.286396 13600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/data/instance:
uuid: "546dbae7fc694683ac1afafaf0e13ab1"
format_stamp: "Formatted at 2026-09-26 16:43:45 on dist-test-slave-wt2j"
server_key: "4b99003cf4eeee890a4d6f8544f5dbc4"
server_key_iv: "e2b2f48d95efc61decccb918cfca75b2"
server_key_version: "encryptionkey@0"
I20260926 16:43:45.287281 13600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance:
uuid: "546dbae7fc694683ac1afafaf0e13ab1"
format_stamp: "Formatted at 2026-09-26 16:43:45 on dist-test-slave-wt2j"
server_key: "4b99003cf4eeee890a4d6f8544f5dbc4"
server_key_iv: "e2b2f48d95efc61decccb918cfca75b2"
server_key_version: "encryptionkey@0"
I20260926 16:43:45.293453 13600 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 16:43:45.297794 13615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:45.300011 13600 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:43:45.300258 13600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "546dbae7fc694683ac1afafaf0e13ab1"
format_stamp: "Formatted at 2026-09-26 16:43:45 on dist-test-slave-wt2j"
server_key: "4b99003cf4eeee890a4d6f8544f5dbc4"
server_key_iv: "e2b2f48d95efc61decccb918cfca75b2"
server_key_version: "encryptionkey@0"
I20260926 16:43:45.300457 13600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:45.332051 13600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:45.334138 13460 heartbeater.cc:503] Master 127.9.238.254:43165 was elected leader, sending a full tablet report...
I20260926 16:43:45.337268 13600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:45.337576 13600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:45.338611 13600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:45.340247 13600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:45.340358 13600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:45.340512 13600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:45.340597 13600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:45.380699 13600 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:38613
I20260926 16:43:45.380733 13727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:38613 every 8 connection(s)
I20260926 16:43:45.382015 13600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:43:45.388356 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 13600
I20260926 16:43:45.388581 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance
I20260926 16:43:45.389067 10171 external_mini_cluster.cc:1428] Setting key 61b32a16dec4c4a3206745af6edff1ee
Sep 26 16:43:45 dist-test-slave-wt2j krb5kdc[13225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441025, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:45.407791 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:45.385781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:35423 (local address 127.9.238.254:43165)
0926 16:43:45.386422 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:45.386431 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:45.387262 (+ 831us) server_negotiation.cc:407] Connection header received
0926 16:43:45.389651 (+ 2389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:45.389658 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:45.389792 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:45.389929 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:45.392168 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:45.393198 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:45.394886 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:45.395399 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:45.395541 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:45.400173 (+ 4632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:45.400180 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:45.400185 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:45.400245 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:45.403394 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:45.404881 (+ 1487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:45.404888 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:45.404894 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:45.405128 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:45.405729 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:45.405735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:45.405739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:45.406157 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:45.406285 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:45.407196 (+ 911us) server_negotiation.cc:299] Negotiation successful
0926 16:43:45.407426 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":312,"threads_started":1}
I20260926 16:43:45.409087 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:45.386125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:43165 (local address 127.9.238.195:35423)
0926 16:43:45.387039 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:45.387095 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:45.389296 (+ 2201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:45.390311 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:45.390325 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:45.391032 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:45.391950 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:45.391980 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:45.393362 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:45.393366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:45.394696 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:45.394709 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:45.395792 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:45.395799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:45.396092 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:45.396877 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:45.396902 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:45.399809 (+ 2907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:45.403700 (+ 3891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:45.403714 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:45.403742 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:45.404632 (+ 890us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:45.405278 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:45.405283 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:45.405287 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:45.405561 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:45.406285 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:45.406292 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:45.406750 (+ 458us) client_negotiation.cc:769] Sending connection context
0926 16:43:45.407022 (+ 272us) client_negotiation.cc:240] Negotiation successful
0926 16:43:45.407527 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":394,"threads_started":1}
I20260926 16:43:45.410880 13728 heartbeater.cc:347] Connected to a master server at 127.9.238.254:43165
I20260926 16:43:45.411240 13728 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:45.412308 13728 heartbeater.cc:511] Master 127.9.238.254:43165 requested a full tablet report, sending...
I20260926 16:43:45.414011 13270 ts_manager.cc:194] Registered new tserver with Master: 546dbae7fc694683ac1afafaf0e13ab1 (127.9.238.195:38613)
I20260926 16:43:45.414880 13270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.195@KRBTEST.COM'} at 127.9.238.195:35423
I20260926 16:43:45.418716 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 16:43:45 dist-test-slave-wt2j krb5kdc[13225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441025, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 16:43:45 dist-test-slave-wt2j krb5kdc[13225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441025, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:45.477260 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:45.461696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47650 (local address 127.9.238.254:43165)
0926 16:43:45.461884 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:45.461890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:45.462519 (+ 629us) server_negotiation.cc:407] Connection header received
0926 16:43:45.462753 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:45.462763 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:45.462833 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:45.462943 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:45.464391 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:45.465144 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:45.466399 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:45.466838 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:45.466982 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:45.471438 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:45.471444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:45.471447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:45.471486 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:45.474527 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:45.475215 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:45.475221 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:45.475224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:45.475355 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:45.475794 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:45.475800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:45.475804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:45.476153 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:45.476399 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:45.476700 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 16:43:45.476927 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 16:43:45.478001 13270 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47650
I20260926 16:43:45.480587 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 13332
Sep 26 16:43:45 dist-test-slave-wt2j krb5kdc[13225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441025, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.238.193@KRBTEST.COM
I20260926 16:43:45.500907 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:45.482167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43186 (local address 127.9.238.193:36013)
0926 16:43:45.482910 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:45.482919 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:45.482946 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:43:45.483114 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:45.483122 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:45.483419 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:45.483647 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:45.485259 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:45.486317 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:45.487938 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:45.488541 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:45.488660 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:45.493670 (+ 5010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:45.493683 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:45.493700 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:45.493739 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:45.496877 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:45.497999 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:45.498009 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:45.498012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:45.498162 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:45.498724 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:45.498730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:45.498733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:45.499333 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:45.499754 (+ 421us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:45.500098 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 16:43:45.500444 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":374,"threads_started":1}
W20260926 16:43:45.502544 13455 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43186
W20260926 16:43:45.504042 10171 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 13332: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:43:45.504287 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 13332
I20260926 16:43:45.523520 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 13466
Sep 26 16:43:45 dist-test-slave-wt2j krb5kdc[13225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441025, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.238.194@KRBTEST.COM
I20260926 16:43:45.541864 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:45.524730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38028 (local address 127.9.238.194:38069)
0926 16:43:45.525327 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:45.525334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:45.525364 (+ 30us) server_negotiation.cc:407] Connection header received
0926 16:43:45.525439 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:45.525446 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:45.525663 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:45.525811 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:45.527203 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:45.528327 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:45.529769 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:45.530386 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:45.530509 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:45.535661 (+ 5152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:45.535672 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:45.535687 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:45.535725 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:45.538828 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:45.539646 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:45.539653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:45.539655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:45.539838 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:45.540325 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:45.540328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:45.540330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:45.540689 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:45.540906 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:45.541330 (+ 424us) server_negotiation.cc:299] Negotiation successful
0926 16:43:45.541547 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":316,"threads_started":1}
W20260926 16:43:45.543864 13589 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38028
W20260926 16:43:45.545092 10171 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 13466: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:43:45.545259 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 13466
I20260926 16:43:45.561539 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 13600
Sep 26 16:43:45 dist-test-slave-wt2j krb5kdc[13225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441025, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.9.238.195@KRBTEST.COM
I20260926 16:43:45.582033 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:45.562640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54858 (local address 127.9.238.195:38613)
0926 16:43:45.563518 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:45.563527 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:45.563641 (+ 114us) server_negotiation.cc:407] Connection header received
0926 16:43:45.563728 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:45.563737 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:45.564049 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:45.564280 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:45.565871 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:45.567043 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:45.568566 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:45.569241 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:45.569405 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:45.574690 (+ 5285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:45.574710 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:45.574729 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:45.574778 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:45.578368 (+ 3590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:45.579709 (+ 1341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:45.579723 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:45.579729 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:45.579888 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:45.580345 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:45.580348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:45.580350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:45.580727 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:45.580965 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:45.581435 (+ 470us) server_negotiation.cc:299] Negotiation successful
0926 16:43:45.581675 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":422,"threads_started":1}
W20260926 16:43:45.583539 13723 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54858
W20260926 16:43:45.584743 10171 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 13600: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:43:45.584921 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 13600
I20260926 16:43:45.601927 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 13238
I20260926 16:43:45.603302 13300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:43:45.710229 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 13238
2026-09-26T16:43:45Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3961 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:43:45 dist-test-slave-wt2j krb5kdc[13746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:43:45 dist-test-slave-wt2j krb5kdc[13746](info): set up 2 sockets
Sep 26 16:43:45 dist-test-slave-wt2j krb5kdc[13746](info): commencing operation
krb5kdc: starting...
W20260926 16:43:46.858202 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:43:47 dist-test-slave-wt2j krb5kdc[13746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:43:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:43:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:43:47.096091 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:37157
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:37157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:47.497066 13759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:47.497480 13759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:47.497654 13759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:47.506295 13759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:43:47.506451 13759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:47.506553 13759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:47.506656 13759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:43:47.506733 13759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:43:47.518293 13759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:37157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:37157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:47.520236 13759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:47.521924 13759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:47.536401 13767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:47.536604 13765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:47.536401 13764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:47.541584 13759 server_base.cc:1034] running on GCE node
I20260926 16:43:47.542693 13759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:47.544677 13759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:47.545990 13759 hybrid_clock.cc:648] HybridClock initialized: now 1790441027545910 us; error 61 us; skew 500 ppm
I20260926 16:43:47.546590 13759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:47 dist-test-slave-wt2j krb5kdc[13746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441027, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:47.555951 13759 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:43:47.558146 13759 webserver.cc:460] Webserver started at http://127.9.238.254:34865/ using document root <none> and password file <none>
I20260926 16:43:47.559126 13759 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:47.559284 13759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:47.559760 13759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:47.563500 13759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "2d5da6267b8c46f882cab0e129796ec4"
format_stamp: "Formatted at 2026-09-26 16:43:47 on dist-test-slave-wt2j"
server_key: "f4969da6720257e9f75fd8ad48106bda"
server_key_iv: "0e9fa8e577006018eada3a9a29a1af0d"
server_key_version: "encryptionkey@0"
I20260926 16:43:47.564584 13759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "2d5da6267b8c46f882cab0e129796ec4"
format_stamp: "Formatted at 2026-09-26 16:43:47 on dist-test-slave-wt2j"
server_key: "f4969da6720257e9f75fd8ad48106bda"
server_key_iv: "0e9fa8e577006018eada3a9a29a1af0d"
server_key_version: "encryptionkey@0"
I20260926 16:43:47.571611 13759 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260926 16:43:47.576354 13774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:47.578299 13759 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260926 16:43:47.578540 13759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "2d5da6267b8c46f882cab0e129796ec4"
format_stamp: "Formatted at 2026-09-26 16:43:47 on dist-test-slave-wt2j"
server_key: "f4969da6720257e9f75fd8ad48106bda"
server_key_iv: "0e9fa8e577006018eada3a9a29a1af0d"
server_key_version: "encryptionkey@0"
I20260926 16:43:47.578814 13759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:47.618587 13759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:47.624159 13759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:47.624554 13759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:47.647811 13759 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:37157
I20260926 16:43:47.647864 13825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:37157 every 8 connection(s)
I20260926 16:43:47.649682 13759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:43:47.652508 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 13759
I20260926 16:43:47.652765 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:43:47.653546 10171 external_mini_cluster.cc:1428] Setting key debcb78c58287dc3dd75f287623a41f0
I20260926 16:43:47.654891 13826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 16:43:47 dist-test-slave-wt2j krb5kdc[13746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:47.671396 13826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d5da6267b8c46f882cab0e129796ec4: Bootstrap starting.
I20260926 16:43:47.676015 13826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d5da6267b8c46f882cab0e129796ec4: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:47.677310 13826 log.cc:824] T 00000000000000000000000000000000 P 2d5da6267b8c46f882cab0e129796ec4: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:47.681475 13826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d5da6267b8c46f882cab0e129796ec4: No bootstrap required, opened a new log
I20260926 16:43:47.681569 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:47.656106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38412 (local address 127.9.238.254:37157)
0926 16:43:47.657085 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:47.657100 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:47.657153 (+ 53us) server_negotiation.cc:407] Connection header received
0926 16:43:47.658916 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:47.658957 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:47.659494 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:47.660031 (+ 537us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:47.662027 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:47.663711 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:47.665499 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:47.666346 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:47.666565 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:47.672030 (+ 5465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:47.672044 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:47.672069 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:47.672140 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:47.676380 (+ 4240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:47.677318 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:47.677329 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:47.677339 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:47.677557 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:47.678177 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:47.678184 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:47.678189 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:47.678822 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:47.679075 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:47.679715 (+ 640us) server_negotiation.cc:299] Negotiation successful
0926 16:43:47.680192 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":434,"threads_started":1}
I20260926 16:43:47.691097 13826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d5da6267b8c46f882cab0e129796ec4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5da6267b8c46f882cab0e129796ec4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 37157 } }
I20260926 16:43:47.691608 13826 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d5da6267b8c46f882cab0e129796ec4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:47.691815 13826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d5da6267b8c46f882cab0e129796ec4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d5da6267b8c46f882cab0e129796ec4, State: Initialized, Role: FOLLOWER
I20260926 16:43:47.692680 13826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d5da6267b8c46f882cab0e129796ec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5da6267b8c46f882cab0e129796ec4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 37157 } }
I20260926 16:43:47.692974 13826 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d5da6267b8c46f882cab0e129796ec4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:43:47.693156 13826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d5da6267b8c46f882cab0e129796ec4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:43:47.693352 13826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d5da6267b8c46f882cab0e129796ec4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:47.695874 13826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d5da6267b8c46f882cab0e129796ec4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5da6267b8c46f882cab0e129796ec4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 37157 } }
I20260926 16:43:47.696517 13826 leader_election.cc:302] T 00000000000000000000000000000000 P 2d5da6267b8c46f882cab0e129796ec4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d5da6267b8c46f882cab0e129796ec4; no voters:
I20260926 16:43:47.697326 13826 leader_election.cc:288] T 00000000000000000000000000000000 P 2d5da6267b8c46f882cab0e129796ec4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:43:47.697602 13831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d5da6267b8c46f882cab0e129796ec4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:47.699204 13831 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d5da6267b8c46f882cab0e129796ec4 [term 1 LEADER]: Becoming Leader. State: Replica: 2d5da6267b8c46f882cab0e129796ec4, State: Running, Role: LEADER
I20260926 16:43:47.700105 13831 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d5da6267b8c46f882cab0e129796ec4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5da6267b8c46f882cab0e129796ec4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 37157 } }
I20260926 16:43:47.701321 13826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d5da6267b8c46f882cab0e129796ec4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:43:47.704373 13832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d5da6267b8c46f882cab0e129796ec4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d5da6267b8c46f882cab0e129796ec4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5da6267b8c46f882cab0e129796ec4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 37157 } } }
I20260926 16:43:47.704864 13832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d5da6267b8c46f882cab0e129796ec4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:47.706957 13836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:43:47.706851 13833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d5da6267b8c46f882cab0e129796ec4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d5da6267b8c46f882cab0e129796ec4. Latest consensus state: current_term: 1 leader_uuid: "2d5da6267b8c46f882cab0e129796ec4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5da6267b8c46f882cab0e129796ec4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 37157 } } }
I20260926 16:43:47.707464 13833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d5da6267b8c46f882cab0e129796ec4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:47.722219 13836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:43:47.733891 13836 catalog_manager.cc:1347] Generated new cluster ID: 1e27ee00705e491c937faaa3ff839efc
I20260926 16:43:47.734040 13836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:43:47.758765 13836 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:43:47.760766 13836 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:43:47.769663 13836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d5da6267b8c46f882cab0e129796ec4: Generated new TSK 0
I20260926 16:43:47.771242 13836 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
I20260926 16:43:47.840494 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:0
--local_ip_for_outbound_sockets=127.9.238.193
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:37157
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:48.263823 13852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:48.264209 13852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:48.264405 13852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:48.273605 13852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:48.273769 13852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:48.273981 13852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:43:48.285447 13852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:37157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:48.287258 13852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:48.289112 13852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:48.306500 13857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:48.306654 13860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:48.306596 13858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:48.308018 13852 server_base.cc:1034] running on GCE node
I20260926 16:43:48.308961 13852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:48.310322 13852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:48.311816 13852 hybrid_clock.cc:648] HybridClock initialized: now 1790441028311741 us; error 40 us; skew 500 ppm
I20260926 16:43:48.312295 13852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:48 dist-test-slave-wt2j krb5kdc[13746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441028, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:48.316882 13852 init.cc:377] Logged in from keytab as kudu/127.9.238.193@KRBTEST.COM (short username kudu)
I20260926 16:43:48.319149 13852 webserver.cc:460] Webserver started at http://127.9.238.193:42027/ using document root <none> and password file <none>
I20260926 16:43:48.320238 13852 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:48.320371 13852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:48.320729 13852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:48.324919 13852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-0/data/instance:
uuid: "e770d6e1249148f9ae11b6e0de1e2c97"
format_stamp: "Formatted at 2026-09-26 16:43:48 on dist-test-slave-wt2j"
server_key: "4b929e3bc042acf089aacb3c9e81302d"
server_key_iv: "d2fbdb5b8818b45b365cdffcc4936c88"
server_key_version: "encryptionkey@0"
I20260926 16:43:48.326049 13852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance:
uuid: "e770d6e1249148f9ae11b6e0de1e2c97"
format_stamp: "Formatted at 2026-09-26 16:43:48 on dist-test-slave-wt2j"
server_key: "4b929e3bc042acf089aacb3c9e81302d"
server_key_iv: "d2fbdb5b8818b45b365cdffcc4936c88"
server_key_version: "encryptionkey@0"
I20260926 16:43:48.333660 13852 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260926 16:43:48.338766 13867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:48.340790 13852 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:43:48.341053 13852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "e770d6e1249148f9ae11b6e0de1e2c97"
format_stamp: "Formatted at 2026-09-26 16:43:48 on dist-test-slave-wt2j"
server_key: "4b929e3bc042acf089aacb3c9e81302d"
server_key_iv: "d2fbdb5b8818b45b365cdffcc4936c88"
server_key_version: "encryptionkey@0"
I20260926 16:43:48.341351 13852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:48.362696 13852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:48.368104 13852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:48.368466 13852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:48.369696 13852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:48.371381 13852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:48.371495 13852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:48.371718 13852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:48.371798 13852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:48.413589 13852 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:38581
I20260926 16:43:48.413606 13979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:38581 every 8 connection(s)
I20260926 16:43:48.415251 13852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:43:48.418646 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 13852
I20260926 16:43:48.418932 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance
I20260926 16:43:48.419836 10171 external_mini_cluster.cc:1428] Setting key 61b8b411ea6886daa380e116b4ab1a07
Sep 26 16:43:48 dist-test-slave-wt2j krb5kdc[13746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441028, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:48.442909 13982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:48.418700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:55059 (local address 127.9.238.254:37157)
0926 16:43:48.419446 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:48.419454 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:48.420926 (+ 1472us) server_negotiation.cc:407] Connection header received
0926 16:43:48.423439 (+ 2513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:48.423444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:48.423534 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:48.423775 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:48.426197 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:48.427114 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:48.429716 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:48.430236 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:48.430394 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:48.435041 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:48.435047 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:48.435104 (+ 57us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:48.435146 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:48.437935 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:48.439744 (+ 1809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:48.439750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:48.439752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:48.439904 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:48.440664 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:48.440668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:48.440670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:48.440998 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:48.441136 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:48.442419 (+ 1283us) server_negotiation.cc:299] Negotiation successful
0926 16:43:48.442636 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":352,"threads_started":1}
WARNING: no policy specified for kudu/127.9.238.194@KRBTEST.COM; defaulting to no policy
I20260926 16:43:48.444898 13983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:48.419629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:37157 (local address 127.9.238.193:55059)
0926 16:43:48.420722 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:48.420786 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:48.423046 (+ 2260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:48.424102 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:48.424119 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:48.425018 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:48.425955 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:48.425983 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:48.427310 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:48.427316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:48.429501 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:48.429535 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:48.430527 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:48.430538 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:48.430786 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:48.431645 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:48.431675 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:48.434657 (+ 2982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:48.438187 (+ 3530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:48.438205 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:48.438248 (+ 43us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:48.439191 (+ 943us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:48.440057 (+ 866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:48.440065 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:48.440071 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:48.440444 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:48.441160 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:48.441172 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:48.441856 (+ 684us) client_negotiation.cc:769] Sending connection context
0926 16:43:48.442285 (+ 429us) client_negotiation.cc:240] Negotiation successful
0926 16:43:48.443093 (+ 808us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":434,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.194@KRBTEST.COM" created.
I20260926 16:43:48.447343 13980 heartbeater.cc:347] Connected to a master server at 127.9.238.254:37157
I20260926 16:43:48.447836 13980 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:48.448971 13980 heartbeater.cc:511] Master 127.9.238.254:37157 requested a full tablet report, sending...
I20260926 16:43:48.452229 13791 ts_manager.cc:194] Registered new tserver with Master: e770d6e1249148f9ae11b6e0de1e2c97 (127.9.238.193:38581)
I20260926 16:43:48.454891 13791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.193@KRBTEST.COM'} at 127.9.238.193:55059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
I20260926 16:43:48.480899 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:0
--local_ip_for_outbound_sockets=127.9.238.194
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:37157
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:48.850373 13986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:48.850689 13986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:48.850821 13986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:48.858291 13986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:48.858418 13986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:48.858599 13986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:43:48.868561 13986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:37157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:48.870011 13986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:48.871389 13986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:48.887080 13992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:48.887100 13991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:48.887184 13994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:48.888837 13986 server_base.cc:1034] running on GCE node
I20260926 16:43:48.889746 13986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:48.891347 13986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:48.892657 13986 hybrid_clock.cc:648] HybridClock initialized: now 1790441028892639 us; error 116 us; skew 500 ppm
I20260926 16:43:48.893090 13986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:48 dist-test-slave-wt2j krb5kdc[13746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441028, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:48.897217 13986 init.cc:377] Logged in from keytab as kudu/127.9.238.194@KRBTEST.COM (short username kudu)
I20260926 16:43:48.899219 13986 webserver.cc:460] Webserver started at http://127.9.238.194:38699/ using document root <none> and password file <none>
I20260926 16:43:48.900229 13986 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:48.900424 13986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:48.900846 13986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:48.904760 13986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-1/data/instance:
uuid: "685683cd53964af09f1b3071e575b9c7"
format_stamp: "Formatted at 2026-09-26 16:43:48 on dist-test-slave-wt2j"
server_key: "137d8d610166abde50ef480ba6c4ed02"
server_key_iv: "0b6795c47f97e535450c9ea7e91969b2"
server_key_version: "encryptionkey@0"
I20260926 16:43:48.905668 13986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance:
uuid: "685683cd53964af09f1b3071e575b9c7"
format_stamp: "Formatted at 2026-09-26 16:43:48 on dist-test-slave-wt2j"
server_key: "137d8d610166abde50ef480ba6c4ed02"
server_key_iv: "0b6795c47f97e535450c9ea7e91969b2"
server_key_version: "encryptionkey@0"
I20260926 16:43:48.912495 13986 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260926 16:43:48.917515 14001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:48.919648 13986 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.007s sys 0.000s
I20260926 16:43:48.919859 13986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "685683cd53964af09f1b3071e575b9c7"
format_stamp: "Formatted at 2026-09-26 16:43:48 on dist-test-slave-wt2j"
server_key: "137d8d610166abde50ef480ba6c4ed02"
server_key_iv: "0b6795c47f97e535450c9ea7e91969b2"
server_key_version: "encryptionkey@0"
I20260926 16:43:48.920156 13986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:48.939240 13986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:48.944962 13986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:48.945312 13986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:48.946839 13986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:48.948728 13986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:48.948828 13986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:48.948953 13986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:48.949018 13986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:48.991313 13986 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:40719
I20260926 16:43:48.991384 14113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:40719 every 8 connection(s)
I20260926 16:43:48.993027 13986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:43:48.997149 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 13986
I20260926 16:43:48.997356 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance
I20260926 16:43:48.997834 10171 external_mini_cluster.cc:1428] Setting key 3957a74b2b4c81f47ac562218ceec728
Sep 26 16:43:49 dist-test-slave-wt2j krb5kdc[13746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441028, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.238.195@KRBTEST.COM; defaulting to no policy
I20260926 16:43:49.023307 14116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:48.996370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:42035 (local address 127.9.238.254:37157)
0926 16:43:48.997452 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:48.997460 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:48.998774 (+ 1314us) server_negotiation.cc:407] Connection header received
0926 16:43:49.001545 (+ 2771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:49.001550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:49.001644 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:49.001766 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:49.004828 (+ 3062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:49.005784 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:49.008813 (+ 3029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:49.009662 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:49.009841 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:49.014787 (+ 4946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:49.014797 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:49.014804 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:49.014863 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:49.018892 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:49.020162 (+ 1270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:49.020169 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:49.020173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:49.020347 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:49.020892 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:49.020899 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:49.020903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:49.021312 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:49.021489 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:49.022501 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0926 16:43:49.022800 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":919,"thread_start_us":755,"threads_started":1}
I20260926 16:43:49.024794 14117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:48.997313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:37157 (local address 127.9.238.194:42035)
0926 16:43:48.998379 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:48.998447 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:49.000970 (+ 2523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:49.002220 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:49.002239 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:49.003334 (+ 1095us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:49.004538 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:49.004578 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:49.006013 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:49.006022 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:49.008593 (+ 2571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:49.008611 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:49.009973 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:49.009978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:49.010239 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:49.011161 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:49.011190 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:49.014418 (+ 3228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:49.019185 (+ 4767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:49.019195 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:49.019216 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:49.019925 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:49.020514 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:49.020518 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:49.020521 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:49.020742 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:49.021457 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:49.021464 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:49.021953 (+ 489us) client_negotiation.cc:769] Sending connection context
0926 16:43:49.022309 (+ 356us) client_negotiation.cc:240] Negotiation successful
0926 16:43:49.023046 (+ 737us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":457,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.195@KRBTEST.COM" created.
I20260926 16:43:49.026948 14114 heartbeater.cc:347] Connected to a master server at 127.9.238.254:37157
I20260926 16:43:49.027441 14114 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:49.028611 14114 heartbeater.cc:511] Master 127.9.238.254:37157 requested a full tablet report, sending...
I20260926 16:43:49.030629 13791 ts_manager.cc:194] Registered new tserver with Master: 685683cd53964af09f1b3071e575b9c7 (127.9.238.194:40719)
I20260926 16:43:49.031930 13791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.194@KRBTEST.COM'} at 127.9.238.194:42035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
I20260926 16:43:49.055970 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:0
--local_ip_for_outbound_sockets=127.9.238.195
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:37157
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:49.389191 14120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:49.389497 14120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:49.389657 14120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:49.396997 14120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:49.397128 14120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:49.397316 14120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:43:49.407691 14120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:37157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:49.409083 14120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:49.410442 14120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:49.427646 14126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:49.427675 14128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:49.427762 14125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:49.434222 14120 server_base.cc:1034] running on GCE node
I20260926 16:43:49.435133 14120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:49.436393 14120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:49.437708 14120 hybrid_clock.cc:648] HybridClock initialized: now 1790441029437632 us; error 60 us; skew 500 ppm
I20260926 16:43:49.438158 14120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:49 dist-test-slave-wt2j krb5kdc[13746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441029, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:49.442221 14120 init.cc:377] Logged in from keytab as kudu/127.9.238.195@KRBTEST.COM (short username kudu)
I20260926 16:43:49.444326 14120 webserver.cc:460] Webserver started at http://127.9.238.195:34829/ using document root <none> and password file <none>
I20260926 16:43:49.445152 14120 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:49.445271 14120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:49.445567 14120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:49.449047 14120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-2/data/instance:
uuid: "8da7b1b6bbf7494ba9ec352fa93dec61"
format_stamp: "Formatted at 2026-09-26 16:43:49 on dist-test-slave-wt2j"
server_key: "e6196bd0298db6dcec49179a9c954234"
server_key_iv: "8198173dad6f4f43cbf5196d61ba6f59"
server_key_version: "encryptionkey@0"
I20260926 16:43:49.449985 14120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance:
uuid: "8da7b1b6bbf7494ba9ec352fa93dec61"
format_stamp: "Formatted at 2026-09-26 16:43:49 on dist-test-slave-wt2j"
server_key: "e6196bd0298db6dcec49179a9c954234"
server_key_iv: "8198173dad6f4f43cbf5196d61ba6f59"
server_key_version: "encryptionkey@0"
I20260926 16:43:49.457180 14120 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260926 16:43:49.459441 13980 heartbeater.cc:503] Master 127.9.238.254:37157 was elected leader, sending a full tablet report...
I20260926 16:43:49.461750 14135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:49.463742 14120 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260926 16:43:49.463970 14120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "8da7b1b6bbf7494ba9ec352fa93dec61"
format_stamp: "Formatted at 2026-09-26 16:43:49 on dist-test-slave-wt2j"
server_key: "e6196bd0298db6dcec49179a9c954234"
server_key_iv: "8198173dad6f4f43cbf5196d61ba6f59"
server_key_version: "encryptionkey@0"
I20260926 16:43:49.464197 14120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:49.486663 14120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:49.492281 14120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:49.492568 14120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:49.493669 14120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:49.495134 14120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:49.495221 14120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:49.495313 14120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:49.495366 14120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:49.536903 14120 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:33027
I20260926 16:43:49.537025 14247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:33027 every 8 connection(s)
I20260926 16:43:49.538391 14120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:43:49.540999 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 14120
I20260926 16:43:49.541245 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance
I20260926 16:43:49.541723 10171 external_mini_cluster.cc:1428] Setting key cc3341fa03a79cf6c6633db0b6bf681e
Sep 26 16:43:49 dist-test-slave-wt2j krb5kdc[13746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441029, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:49.564483 14250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:49.542220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:50785 (local address 127.9.238.254:37157)
0926 16:43:49.542839 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:49.542844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:49.544422 (+ 1578us) server_negotiation.cc:407] Connection header received
0926 16:43:49.547025 (+ 2603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:49.547031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:49.547135 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:49.547248 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:49.549346 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:49.550296 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:49.552537 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:49.552958 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:49.553108 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:49.557636 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:49.557642 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:49.557645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:49.557693 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:49.560857 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:49.561748 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:49.561761 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:49.561764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:49.561900 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:49.562391 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:49.562400 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:49.562402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:49.562702 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:49.562878 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:49.563778 (+ 900us) server_negotiation.cc:299] Negotiation successful
0926 16:43:49.564036 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":346,"threads_started":1}
I20260926 16:43:49.565757 14251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:49.542883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:37157 (local address 127.9.238.195:50785)
0926 16:43:49.544163 (+ 1280us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:49.544245 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:49.546666 (+ 2421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:49.547618 (+ 952us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:49.547630 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:49.548354 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:49.549147 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:49.549168 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:49.550491 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:49.550497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:49.552375 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:49.552391 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:49.553220 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:49.553226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:49.553487 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:49.554334 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:49.554366 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:49.557376 (+ 3010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:49.561042 (+ 3666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:49.561054 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:49.561080 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:49.561602 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:49.562018 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:49.562024 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:49.562028 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:49.562257 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:49.562837 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:49.562848 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:49.563314 (+ 466us) client_negotiation.cc:769] Sending connection context
0926 16:43:49.563666 (+ 352us) client_negotiation.cc:240] Negotiation successful
0926 16:43:49.564345 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":386,"threads_started":1}
I20260926 16:43:49.567649 14248 heartbeater.cc:347] Connected to a master server at 127.9.238.254:37157
I20260926 16:43:49.568048 14248 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:49.569046 14248 heartbeater.cc:511] Master 127.9.238.254:37157 requested a full tablet report, sending...
I20260926 16:43:49.571175 13791 ts_manager.cc:194] Registered new tserver with Master: 8da7b1b6bbf7494ba9ec352fa93dec61 (127.9.238.195:33027)
I20260926 16:43:49.572438 13791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.195@KRBTEST.COM'} at 127.9.238.195:50785
I20260926 16:43:49.572707 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 16:43:49 dist-test-slave-wt2j krb5kdc[13746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 16:43:49 dist-test-slave-wt2j krb5kdc[13746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.193@KRBTEST.COM
I20260926 16:43:49.632808 14256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:49.615012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56518 (local address 127.9.238.193:38581)
0926 16:43:49.615946 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:49.615954 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:49.615987 (+ 33us) server_negotiation.cc:407] Connection header received
0926 16:43:49.616065 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:49.616072 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:49.616305 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:49.616453 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:49.618164 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:49.619354 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:49.621100 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:49.621735 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:49.621858 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:49.626568 (+ 4710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:49.626582 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:49.626597 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:49.626640 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:49.629690 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:49.630595 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:49.630602 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:49.630604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:49.630741 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:49.631127 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:49.631131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:49.631132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:49.631474 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:49.631780 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:49.632295 (+ 515us) server_negotiation.cc:299] Negotiation successful
0926 16:43:49.632511 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":510,"threads_started":1}
Sep 26 16:43:49 dist-test-slave-wt2j krb5kdc[13746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:49.658013 14250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:49.639393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38428 (local address 127.9.238.254:37157)
0926 16:43:49.639737 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:49.639743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:49.640009 (+ 266us) server_negotiation.cc:407] Connection header received
0926 16:43:49.640182 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:49.640186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:49.640263 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:49.640367 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:49.642169 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:49.642963 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:49.644700 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:49.645145 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:49.645257 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:49.650702 (+ 5445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:49.650710 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:49.650714 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:49.650771 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:49.654342 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:49.655340 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:49.655352 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:49.655360 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:49.655520 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:49.656042 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:49.656048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:49.656053 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:49.656476 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:49.656799 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:49.657270 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 16:43:49.657620 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
W20260926 16:43:49.658749 13791 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38428
I20260926 16:43:49.660221 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 13852
I20260926 16:43:49.678145 14256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:49.661512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56534 (local address 127.9.238.193:38581)
0926 16:43:49.661833 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:49.661842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:49.662063 (+ 221us) server_negotiation.cc:407] Connection header received
0926 16:43:49.662355 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:49.662362 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:49.662454 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:49.662581 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:49.664548 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:49.665524 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:49.667120 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:49.667755 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:49.667940 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:49.671462 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:49.671470 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:49.671473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:49.671528 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:49.674747 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:49.675835 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:49.675842 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:49.675845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:49.675998 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:49.676427 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:49.676435 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:49.676439 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:49.676763 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:49.677056 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:49.677442 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 16:43:49.677733 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260926 16:43:49.679188 13975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:43:49.767979 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 13852
I20260926 16:43:49.788888 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 13986
Sep 26 16:43:49 dist-test-slave-wt2j krb5kdc[13746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.194@KRBTEST.COM
I20260926 16:43:49.809127 14261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:49.790091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44204 (local address 127.9.238.194:40719)
0926 16:43:49.790804 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:49.790813 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:49.790844 (+ 31us) server_negotiation.cc:407] Connection header received
0926 16:43:49.790909 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:49.790916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:49.791148 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:49.791354 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:49.792929 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:49.794091 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:49.795544 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:49.796290 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:49.796418 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:49.801951 (+ 5533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:49.801969 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:49.801986 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:49.802032 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:49.805467 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:49.806505 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:49.806513 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:49.806515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:49.806676 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:49.807156 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:49.807160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:49.807163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:49.807672 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:49.807935 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:49.808356 (+ 421us) server_negotiation.cc:299] Negotiation successful
0926 16:43:49.808605 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":371,"threads_started":1}
I20260926 16:43:49.811146 14109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:43:49.900501 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 13986
I20260926 16:43:49.919135 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 14120
Sep 26 16:43:49 dist-test-slave-wt2j krb5kdc[13746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.195@KRBTEST.COM
I20260926 16:43:49.939009 14263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:49.920531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38482 (local address 127.9.238.195:33027)
0926 16:43:49.921439 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:49.921448 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:49.921494 (+ 46us) server_negotiation.cc:407] Connection header received
0926 16:43:49.921561 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:49.921569 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:49.921798 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:49.922017 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:49.923369 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:49.924518 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:49.925865 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:49.926436 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:49.926585 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:49.931824 (+ 5239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:49.931845 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:49.931867 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:49.931935 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:49.935334 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:49.936420 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:49.936430 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:49.936433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:49.936618 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:49.937079 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:49.937083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:49.937085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:49.937555 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:49.937850 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:49.938372 (+ 522us) server_negotiation.cc:299] Negotiation successful
0926 16:43:49.938653 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":508,"threads_started":1}
I20260926 16:43:49.940804 14243 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:43:50.039150 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 14120
I20260926 16:43:50.055929 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 13759
I20260926 16:43:50.057396 13821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:43:50.165450 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 13759
2026-09-26T16:43:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4452 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:43:50 dist-test-slave-wt2j krb5kdc[14268](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:43:50 dist-test-slave-wt2j krb5kdc[14268](info): set up 2 sockets
Sep 26 16:43:50 dist-test-slave-wt2j krb5kdc[14268](info): commencing operation
krb5kdc: starting...
W20260926 16:43:51.329797 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.119s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:43:51 dist-test-slave-wt2j krb5kdc[14268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:43:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:43:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:43:51.569495 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:44881
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:44881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:51.914208 14281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:51.914633 14281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:51.914808 14281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:51.922717 14281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:43:51.922900 14281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:51.923038 14281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:51.923153 14281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:43:51.923271 14281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:43:51.935084 14281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:44881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:44881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:51.936604 14281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:51.938064 14281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:51.954177 14289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:51.954286 14287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:51.954241 14286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:51.955096 14281 server_base.cc:1034] running on GCE node
I20260926 16:43:51.956100 14281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:51.957720 14281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:51.959012 14281 hybrid_clock.cc:648] HybridClock initialized: now 1790441031958932 us; error 61 us; skew 500 ppm
I20260926 16:43:51.959498 14281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:51 dist-test-slave-wt2j krb5kdc[14268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441031, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:51.963399 14281 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:43:51.964955 14281 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:51.965062 14281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:51.965466 14281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:51.968830 14281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "500363ef762e47edb00c1552be6f6f4b"
format_stamp: "Formatted at 2026-09-26 16:43:51 on dist-test-slave-wt2j"
server_key: "2c658e85e6610c87fe56784146315539"
server_key_iv: "3528194032ba86e3fb4a9d205b9193a5"
server_key_version: "encryptionkey@0"
I20260926 16:43:51.969725 14281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "500363ef762e47edb00c1552be6f6f4b"
format_stamp: "Formatted at 2026-09-26 16:43:51 on dist-test-slave-wt2j"
server_key: "2c658e85e6610c87fe56784146315539"
server_key_iv: "3528194032ba86e3fb4a9d205b9193a5"
server_key_version: "encryptionkey@0"
I20260926 16:43:51.975883 14281 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260926 16:43:51.980458 14295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:51.982384 14281 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 16:43:51.982681 14281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "500363ef762e47edb00c1552be6f6f4b"
format_stamp: "Formatted at 2026-09-26 16:43:51 on dist-test-slave-wt2j"
server_key: "2c658e85e6610c87fe56784146315539"
server_key_iv: "3528194032ba86e3fb4a9d205b9193a5"
server_key_version: "encryptionkey@0"
I20260926 16:43:51.982964 14281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:52.005491 14281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:52.011910 14281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:52.012362 14281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:52.035079 14281 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:44881
I20260926 16:43:52.035115 14346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:44881 every 8 connection(s)
I20260926 16:43:52.036111 14281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:43:52.041610 14347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:52.045184 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 14281
I20260926 16:43:52.045456 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:43:52.046135 10171 external_mini_cluster.cc:1428] Setting key 064fa4afcc4b26add47c526b6c1b7f13
I20260926 16:43:52.057602 14347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500363ef762e47edb00c1552be6f6f4b: Bootstrap starting.
Sep 26 16:43:52 dist-test-slave-wt2j krb5kdc[14268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:52.062403 14347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 500363ef762e47edb00c1552be6f6f4b: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:52.064630 14347 log.cc:824] T 00000000000000000000000000000000 P 500363ef762e47edb00c1552be6f6f4b: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:52.069677 14347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500363ef762e47edb00c1552be6f6f4b: No bootstrap required, opened a new log
I20260926 16:43:52.075006 14350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:52.048793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40586 (local address 127.9.238.254:44881)
0926 16:43:52.049927 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:52.049943 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:52.049997 (+ 54us) server_negotiation.cc:407] Connection header received
0926 16:43:52.052158 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:52.052190 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:52.052708 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:52.053198 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:52.054825 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:52.056273 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:52.057653 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:52.058468 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:52.058699 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:52.062969 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:52.062990 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:52.063023 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:52.063121 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:52.068401 (+ 5280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:52.069358 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:52.069370 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:52.069383 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:52.069677 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:52.070419 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:52.070430 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:52.070436 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:52.071389 (+ 953us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:52.071830 (+ 441us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:52.072386 (+ 556us) server_negotiation.cc:299] Negotiation successful
0926 16:43:52.073060 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":853,"thread_start_us":519,"threads_started":1}
I20260926 16:43:52.082195 14347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 500363ef762e47edb00c1552be6f6f4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500363ef762e47edb00c1552be6f6f4b" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44881 } }
I20260926 16:43:52.082729 14347 raft_consensus.cc:374] T 00000000000000000000000000000000 P 500363ef762e47edb00c1552be6f6f4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:52.082865 14347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 500363ef762e47edb00c1552be6f6f4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500363ef762e47edb00c1552be6f6f4b, State: Initialized, Role: FOLLOWER
I20260926 16:43:52.083794 14347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 500363ef762e47edb00c1552be6f6f4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500363ef762e47edb00c1552be6f6f4b" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44881 } }
I20260926 16:43:52.084187 14347 raft_consensus.cc:388] T 00000000000000000000000000000000 P 500363ef762e47edb00c1552be6f6f4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:43:52.084375 14347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 500363ef762e47edb00c1552be6f6f4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:43:52.084612 14347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 500363ef762e47edb00c1552be6f6f4b [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:52.088887 14347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 500363ef762e47edb00c1552be6f6f4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500363ef762e47edb00c1552be6f6f4b" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44881 } }
I20260926 16:43:52.089870 14347 leader_election.cc:302] T 00000000000000000000000000000000 P 500363ef762e47edb00c1552be6f6f4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 500363ef762e47edb00c1552be6f6f4b; no voters:
I20260926 16:43:52.090742 14347 leader_election.cc:288] T 00000000000000000000000000000000 P 500363ef762e47edb00c1552be6f6f4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:43:52.090924 14352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 500363ef762e47edb00c1552be6f6f4b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:52.092063 14352 raft_consensus.cc:686] T 00000000000000000000000000000000 P 500363ef762e47edb00c1552be6f6f4b [term 1 LEADER]: Becoming Leader. State: Replica: 500363ef762e47edb00c1552be6f6f4b, State: Running, Role: LEADER
I20260926 16:43:52.092739 14352 consensus_queue.cc:237] T 00000000000000000000000000000000 P 500363ef762e47edb00c1552be6f6f4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500363ef762e47edb00c1552be6f6f4b" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44881 } }
I20260926 16:43:52.093963 14347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 500363ef762e47edb00c1552be6f6f4b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:43:52.096513 14354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500363ef762e47edb00c1552be6f6f4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 500363ef762e47edb00c1552be6f6f4b. Latest consensus state: current_term: 1 leader_uuid: "500363ef762e47edb00c1552be6f6f4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500363ef762e47edb00c1552be6f6f4b" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44881 } } }
I20260926 16:43:52.097417 14354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500363ef762e47edb00c1552be6f6f4b [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:52.098769 14353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500363ef762e47edb00c1552be6f6f4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "500363ef762e47edb00c1552be6f6f4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500363ef762e47edb00c1552be6f6f4b" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44881 } } }
I20260926 16:43:52.099646 14353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500363ef762e47edb00c1552be6f6f4b [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:52.105777 14358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:43:52.111766 14358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:43:52.123405 14358 catalog_manager.cc:1347] Generated new cluster ID: f7c1dbf346d7431a8b03f1280137c2c5
I20260926 16:43:52.123729 14358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:43:52.155057 14358 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:43:52.157174 14358 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:43:52.177279 14358 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 500363ef762e47edb00c1552be6f6f4b: Generated new TSK 0
I20260926 16:43:52.178498 14358 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
I20260926 16:43:52.240392 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:0
--local_ip_for_outbound_sockets=127.9.238.193
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44881
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:52.588680 14373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:52.589083 14373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:52.589248 14373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:52.597707 14373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:52.597854 14373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:52.598026 14373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:43:52.609162 14373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:52.610826 14373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:52.612309 14373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:52.627678 14379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:52.627786 14381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:52.627681 14378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:52.629405 14373 server_base.cc:1034] running on GCE node
I20260926 16:43:52.630180 14373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:52.631290 14373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:52.632548 14373 hybrid_clock.cc:648] HybridClock initialized: now 1790441032632483 us; error 69 us; skew 500 ppm
I20260926 16:43:52.632943 14373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:52 dist-test-slave-wt2j krb5kdc[14268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441032, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:52.646486 14373 init.cc:377] Logged in from keytab as kudu/127.9.238.193@KRBTEST.COM (short username kudu)
I20260926 16:43:52.648079 14373 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:52.648226 14373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:52.648559 14373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:52.651858 14373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-0/data/instance:
uuid: "439b87f2e3be4bd1ba1685f95d6dddc8"
format_stamp: "Formatted at 2026-09-26 16:43:52 on dist-test-slave-wt2j"
server_key: "df6e8317612cc7729ccbe59bf0e2e498"
server_key_iv: "9a6596406f56917faec4233f65040ee9"
server_key_version: "encryptionkey@0"
I20260926 16:43:52.652670 14373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance:
uuid: "439b87f2e3be4bd1ba1685f95d6dddc8"
format_stamp: "Formatted at 2026-09-26 16:43:52 on dist-test-slave-wt2j"
server_key: "df6e8317612cc7729ccbe59bf0e2e498"
server_key_iv: "9a6596406f56917faec4233f65040ee9"
server_key_version: "encryptionkey@0"
I20260926 16:43:52.658494 14373 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:43:52.662873 14387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:52.664897 14373 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:43:52.665341 14373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "439b87f2e3be4bd1ba1685f95d6dddc8"
format_stamp: "Formatted at 2026-09-26 16:43:52 on dist-test-slave-wt2j"
server_key: "df6e8317612cc7729ccbe59bf0e2e498"
server_key_iv: "9a6596406f56917faec4233f65040ee9"
server_key_version: "encryptionkey@0"
I20260926 16:43:52.665627 14373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:52.681106 14373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:52.685762 14373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:52.686072 14373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:52.687281 14373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:52.688941 14373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:52.689069 14373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:52.689205 14373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:52.689307 14373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:52.728971 14373 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:44589
I20260926 16:43:52.728995 14499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:44589 every 8 connection(s)
I20260926 16:43:52.729847 14373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:43:52.732623 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 14373
I20260926 16:43:52.732893 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance
I20260926 16:43:52.733652 10171 external_mini_cluster.cc:1428] Setting key f544a93d4b06ed58b6e1cfb1dac8ceb2
Sep 26 16:43:52 dist-test-slave-wt2j krb5kdc[14268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441032, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:52.754577 14503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:52.734840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:56765 (local address 127.9.238.254:44881)
0926 16:43:52.736489 (+ 1649us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:52.736494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:52.736518 (+ 24us) server_negotiation.cc:407] Connection header received
0926 16:43:52.737218 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:52.737233 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:52.737354 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:52.737539 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:52.739205 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:52.740528 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:52.742343 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:52.743107 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:52.743311 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:52.747841 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:52.747847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:52.747851 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:52.747889 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:52.750892 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:52.751860 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:52.751868 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:52.751876 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:52.752008 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:52.752537 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:52.752548 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:52.752550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:52.752869 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:52.753039 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:52.753890 (+ 851us) server_negotiation.cc:299] Negotiation successful
0926 16:43:52.754150 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1505,"thread_start_us":1372,"threads_started":1}
I20260926 16:43:52.756040 14502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:52.733831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:44881 (local address 127.9.238.193:56765)
0926 16:43:52.734713 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:52.734768 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:52.736869 (+ 2101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:52.737760 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:52.737770 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:52.738344 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:52.738960 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:52.738979 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:52.740765 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:52.740769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:52.742124 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:52.742137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:52.743405 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:52.743410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:52.743731 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:52.744597 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:52.744640 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:52.747508 (+ 2868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:52.751079 (+ 3571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:52.751093 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:52.751117 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:52.751703 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:52.752128 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:52.752133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:52.752137 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:52.752425 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:52.753004 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:52.753014 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:52.753464 (+ 450us) client_negotiation.cc:769] Sending connection context
0926 16:43:52.753785 (+ 321us) client_negotiation.cc:240] Negotiation successful
0926 16:43:52.754421 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":414,"threads_started":1}
WARNING: no policy specified for kudu/127.9.238.194@KRBTEST.COM; defaulting to no policy
I20260926 16:43:52.758260 14500 heartbeater.cc:347] Connected to a master server at 127.9.238.254:44881
I20260926 16:43:52.758692 14500 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.194@KRBTEST.COM" created.
I20260926 16:43:52.759881 14500 heartbeater.cc:511] Master 127.9.238.254:44881 requested a full tablet report, sending...
I20260926 16:43:52.763178 14312 ts_manager.cc:194] Registered new tserver with Master: 439b87f2e3be4bd1ba1685f95d6dddc8 (127.9.238.193:44589)
I20260926 16:43:52.764990 14312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.193@KRBTEST.COM'} at 127.9.238.193:56765
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
I20260926 16:43:52.787806 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:0
--local_ip_for_outbound_sockets=127.9.238.194
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44881
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:53.128841 14506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:53.129148 14506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:53.129244 14506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:53.136751 14506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:53.136878 14506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:53.136996 14506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:43:53.147149 14506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:53.148561 14506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:53.149885 14506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:53.165308 14511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:53.165376 14514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:53.165256 14512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:53.166391 14506 server_base.cc:1034] running on GCE node
I20260926 16:43:53.167119 14506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:53.168349 14506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:53.169620 14506 hybrid_clock.cc:648] HybridClock initialized: now 1790441033169545 us; error 60 us; skew 500 ppm
I20260926 16:43:53.170030 14506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:53 dist-test-slave-wt2j krb5kdc[14268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441033, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:53.173771 14506 init.cc:377] Logged in from keytab as kudu/127.9.238.194@KRBTEST.COM (short username kudu)
I20260926 16:43:53.175169 14506 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:53.175318 14506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:53.175727 14506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:53.178732 14506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-1/data/instance:
uuid: "92a2302d8fb844f796ea23f2481a2407"
format_stamp: "Formatted at 2026-09-26 16:43:53 on dist-test-slave-wt2j"
server_key: "e2ad65e014ae6adc0f2964fd574d394d"
server_key_iv: "448b8e3f675fc6126d5b918bcac358a8"
server_key_version: "encryptionkey@0"
I20260926 16:43:53.179740 14506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance:
uuid: "92a2302d8fb844f796ea23f2481a2407"
format_stamp: "Formatted at 2026-09-26 16:43:53 on dist-test-slave-wt2j"
server_key: "e2ad65e014ae6adc0f2964fd574d394d"
server_key_iv: "448b8e3f675fc6126d5b918bcac358a8"
server_key_version: "encryptionkey@0"
I20260926 16:43:53.185302 14506 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 16:43:53.189512 14520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:53.191251 14506 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260926 16:43:53.191498 14506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "92a2302d8fb844f796ea23f2481a2407"
format_stamp: "Formatted at 2026-09-26 16:43:53 on dist-test-slave-wt2j"
server_key: "e2ad65e014ae6adc0f2964fd574d394d"
server_key_iv: "448b8e3f675fc6126d5b918bcac358a8"
server_key_version: "encryptionkey@0"
I20260926 16:43:53.191874 14506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:53.211534 14506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:53.216284 14506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:53.216631 14506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:53.217774 14506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:53.219334 14506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:53.219463 14506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:53.219633 14506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:53.219728 14506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:53.258144 14506 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:44653
I20260926 16:43:53.258244 14632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:44653 every 8 connection(s)
I20260926 16:43:53.259114 14506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:43:53.262487 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 14506
I20260926 16:43:53.262710 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance
I20260926 16:43:53.263276 10171 external_mini_cluster.cc:1428] Setting key c8874fca3e8440f625034ed77d671367
Sep 26 16:43:53 dist-test-slave-wt2j krb5kdc[14268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441033, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:53.286305 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:53.262454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:53303 (local address 127.9.238.254:44881)
0926 16:43:53.263003 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:53.263011 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:53.264591 (+ 1580us) server_negotiation.cc:407] Connection header received
0926 16:43:53.266918 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:53.266924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:53.267037 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:53.267182 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:53.269272 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:53.270051 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:53.271938 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:53.272445 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:53.272604 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:53.277723 (+ 5119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:53.277734 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:53.277743 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:53.277803 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:53.282017 (+ 4214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:53.283107 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:53.283116 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:53.283122 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:53.283326 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:53.283938 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:53.283946 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:53.283953 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:53.284456 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:53.284660 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:53.285479 (+ 819us) server_negotiation.cc:299] Negotiation successful
0926 16:43:53.285782 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":287,"threads_started":1}
I20260926 16:43:53.287014 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:53.263187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:44881 (local address 127.9.238.194:53303)
0926 16:43:53.264368 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:53.264455 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:53.266576 (+ 2121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:53.267474 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:53.267485 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:53.268255 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:53.269038 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:53.269059 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:53.270228 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:53.270233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:53.271769 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:53.271783 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:53.272723 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:53.272730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:53.273118 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:53.274019 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:53.274059 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:53.277339 (+ 3280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:53.282277 (+ 4938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:53.282287 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:53.282305 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:53.282906 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:53.283472 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:53.283475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:53.283478 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:53.283772 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:53.284608 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:53.284630 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:53.285015 (+ 385us) client_negotiation.cc:769] Sending connection context
0926 16:43:53.285285 (+ 270us) client_negotiation.cc:240] Negotiation successful
0926 16:43:53.285763 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"thread_start_us":486,"threads_started":1}
WARNING: no policy specified for kudu/127.9.238.195@KRBTEST.COM; defaulting to no policy
I20260926 16:43:53.288825 14633 heartbeater.cc:347] Connected to a master server at 127.9.238.254:44881
I20260926 16:43:53.289353 14633 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:53.290439 14633 heartbeater.cc:511] Master 127.9.238.254:44881 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.195@KRBTEST.COM" created.
I20260926 16:43:53.293025 14312 ts_manager.cc:194] Registered new tserver with Master: 92a2302d8fb844f796ea23f2481a2407 (127.9.238.194:44653)
I20260926 16:43:53.294507 14312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.194@KRBTEST.COM'} at 127.9.238.194:53303
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
I20260926 16:43:53.324194 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:0
--local_ip_for_outbound_sockets=127.9.238.195
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44881
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:39483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:43:53.750583 14639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:53.751188 14639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:53.751441 14639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:53.762173 14639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:53.762332 14639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:53.762555 14639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:43:53.769336 14500 heartbeater.cc:503] Master 127.9.238.254:44881 was elected leader, sending a full tablet report...
I20260926 16:43:53.774883 14639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:39483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:53.776754 14639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:53.778438 14639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:53.795293 14645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:53.795281 14644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:53.795697 14647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:53.796561 14639 server_base.cc:1034] running on GCE node
I20260926 16:43:53.797382 14639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:53.798661 14639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:53.799942 14639 hybrid_clock.cc:648] HybridClock initialized: now 1790441033799850 us; error 95 us; skew 500 ppm
I20260926 16:43:53.800454 14639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:43:53 dist-test-slave-wt2j krb5kdc[14268](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441033, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:43:53.804888 14639 init.cc:377] Logged in from keytab as kudu/127.9.238.195@KRBTEST.COM (short username kudu)
I20260926 16:43:53.806561 14639 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:53.806713 14639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:53.807101 14639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:53.810864 14639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-2/data/instance:
uuid: "cbc1dd7520344f4c8e82af61ceb7a51e"
format_stamp: "Formatted at 2026-09-26 16:43:53 on dist-test-slave-wt2j"
server_key: "de6f2b45d93b6392dfe4061ee1a11561"
server_key_iv: "db550776bce3a09c6c1ab73cf397bea5"
server_key_version: "encryptionkey@0"
I20260926 16:43:53.812005 14639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance:
uuid: "cbc1dd7520344f4c8e82af61ceb7a51e"
format_stamp: "Formatted at 2026-09-26 16:43:53 on dist-test-slave-wt2j"
server_key: "de6f2b45d93b6392dfe4061ee1a11561"
server_key_iv: "db550776bce3a09c6c1ab73cf397bea5"
server_key_version: "encryptionkey@0"
I20260926 16:43:53.818542 14639 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 16:43:53.823720 14653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:53.825810 14639 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:43:53.826102 14639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "cbc1dd7520344f4c8e82af61ceb7a51e"
format_stamp: "Formatted at 2026-09-26 16:43:53 on dist-test-slave-wt2j"
server_key: "de6f2b45d93b6392dfe4061ee1a11561"
server_key_iv: "db550776bce3a09c6c1ab73cf397bea5"
server_key_version: "encryptionkey@0"
I20260926 16:43:53.826419 14639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:53.858440 14639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:53.863749 14639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:53.864064 14639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:53.865164 14639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:53.866848 14639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:53.866997 14639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:53.867133 14639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:53.867224 14639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:53.909423 14639 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:35983
I20260926 16:43:53.909598 14765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:35983 every 8 connection(s)
I20260926 16:43:53.910451 14639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:43:53.920250 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 14639
I20260926 16:43:53.920591 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance
I20260926 16:43:53.921088 10171 external_mini_cluster.cc:1428] Setting key f445016ff31149b8f5ce2c34cb8b3f4b
Sep 26 16:43:53 dist-test-slave-wt2j krb5kdc[14268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441033, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:43:53.932698 14768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:53.913765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:54207 (local address 127.9.238.254:44881)
0926 16:43:53.914285 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:53.914290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:53.915488 (+ 1198us) server_negotiation.cc:407] Connection header received
0926 16:43:53.917565 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:53.917570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:53.917691 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:53.917815 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:53.919486 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:53.920348 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:53.922223 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:53.922635 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:53.922789 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:53.926556 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:53.926569 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:53.926572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:53.926606 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:53.929359 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:53.930105 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:53.930113 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:53.930118 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:53.930241 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:53.930665 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:53.930670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:53.930673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:53.931025 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:53.931247 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:53.932057 (+ 810us) server_negotiation.cc:299] Negotiation successful
0926 16:43:53.932287 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":281,"threads_started":1}
I20260926 16:43:53.933727 14769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:53.914390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:44881 (local address 127.9.238.195:54207)
0926 16:43:53.915294 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:53.915344 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:53.917256 (+ 1912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:53.918082 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:53.918091 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:53.918662 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:53.919291 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:53.919308 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:53.920504 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:53.920512 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:53.922077 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:53.922090 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:53.922893 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:53.922901 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:53.923148 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:53.923962 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:43:53.923990 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:53.926308 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:53.929515 (+ 3207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:53.929524 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:53.929542 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:53.929975 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:53.930350 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:43:53.930357 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:43:53.930360 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:43:53.930544 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:43:53.931204 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:53.931215 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:53.931672 (+ 457us) client_negotiation.cc:769] Sending connection context
0926 16:43:53.931968 (+ 296us) client_negotiation.cc:240] Negotiation successful
0926 16:43:53.932576 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":350,"threads_started":1}
I20260926 16:43:53.935535 14766 heartbeater.cc:347] Connected to a master server at 127.9.238.254:44881
I20260926 16:43:53.936123 14766 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:53.937323 14766 heartbeater.cc:511] Master 127.9.238.254:44881 requested a full tablet report, sending...
I20260926 16:43:53.939123 14312 ts_manager.cc:194] Registered new tserver with Master: cbc1dd7520344f4c8e82af61ceb7a51e (127.9.238.195:35983)
I20260926 16:43:53.940070 14312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.195@KRBTEST.COM'} at 127.9.238.195:54207
I20260926 16:43:53.950567 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:43:53.971136 14768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:53.956040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40598 (local address 127.9.238.254:44881)
0926 16:43:53.956260 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:53.956266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:53.956832 (+ 566us) server_negotiation.cc:407] Connection header received
0926 16:43:53.956999 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:53.957003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:53.957100 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:53.957232 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:53.958696 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:53.959677 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:53.961023 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:53.961539 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:53.961660 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:53.964207 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:53.964213 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:53.964217 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:53.964275 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:53.967620 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:53.968536 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:53.968542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:53.968546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:53.968684 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:53.969125 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:53.969131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:53.969135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:53.969598 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:53.969896 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:53.970354 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 16:43:53.970628 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 16:43:53.976136 14312 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:43:53.983292 14312 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:43:54.017882 14783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:54.001046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53188 (local address 127.9.238.193:44589)
0926 16:43:54.008453 (+ 7407us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:54.008463 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:54.008888 (+ 425us) server_negotiation.cc:407] Connection header received
0926 16:43:54.008995 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:54.009003 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:54.009313 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:54.009574 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:54.010654 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.012250 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.015020 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.016342 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.016489 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:54.016761 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:54.017347 (+ 586us) server_negotiation.cc:299] Negotiation successful
0926 16:43:54.017510 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7007,"thread_start_us":6834,"threads_started":1}
I20260926 16:43:54.018203 14780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:54.000958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:44589 (local address 127.0.0.1:53188)
0926 16:43:54.008629 (+ 7671us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:54.008646 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:54.008765 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:54.009895 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:54.009904 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:54.009938 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:54.010409 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:54.010424 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.012402 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.012410 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:54.014868 (+ 2458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:54.014879 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.016870 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.016874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:54.017027 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:54.017154 (+ 127us) client_negotiation.cc:769] Sending connection context
0926 16:43:54.017305 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 16:43:54.017511 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7553,"mutex_wait_us":71,"thread_start_us":5986,"threads_started":1}
I20260926 16:43:54.020421 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:54.000868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45998 (local address 127.9.238.194:44653)
0926 16:43:54.006940 (+ 6072us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:54.006953 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:54.006996 (+ 43us) server_negotiation.cc:407] Connection header received
0926 16:43:54.008178 (+ 1182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:54.008191 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:54.008615 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:54.008867 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:54.011069 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.013396 (+ 2327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.016671 (+ 3275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.018749 (+ 2078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.018911 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:54.019174 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:54.019870 (+ 696us) server_negotiation.cc:299] Negotiation successful
0926 16:43:54.020072 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5898,"thread_start_us":5588,"threads_started":1}
I20260926 16:43:54.020419 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:53.999212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:44653 (local address 127.0.0.1:45998)
0926 16:43:54.006086 (+ 6874us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:54.006151 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:54.007771 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:54.009269 (+ 1498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:54.009278 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:54.009375 (+ 97us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:54.010697 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:54.010717 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.013600 (+ 2883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.013611 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:54.016431 (+ 2820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:54.016451 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.019374 (+ 2923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.019382 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:54.019624 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:54.019667 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 16:43:54.019800 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 16:43:54.019971 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6487,"thread_start_us":6280,"threads_started":1}
I20260926 16:43:54.022531 14435 tablet_service.cc:1467] Processing CreateTablet for tablet 859f1dc2a56c4464b2ab26602702c5d4 (DEFAULT_TABLE table=test-table [id=4019732aa2bb482fba12a7de1390bab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:54.023777 14784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:54.009302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57854 (local address 127.9.238.195:35983)
0926 16:43:54.011913 (+ 2611us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:54.011924 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:54.011964 (+ 40us) server_negotiation.cc:407] Connection header received
0926 16:43:54.012760 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:54.012771 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:54.013103 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:54.013352 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:54.014306 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.016265 (+ 1959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.019729 (+ 3464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.021284 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.021433 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:54.021666 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:54.023072 (+ 1406us) server_negotiation.cc:299] Negotiation successful
0926 16:43:54.023320 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2360,"thread_start_us":3075,"threads_started":1}
I20260926 16:43:54.023808 14782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:54.000379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:35983 (local address 127.0.0.1:57854)
0926 16:43:54.007485 (+ 7106us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:54.007511 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:54.007728 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:54.013344 (+ 5616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:54.013350 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:54.013381 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:54.014024 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:54.014038 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.016489 (+ 2451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.016498 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:54.019460 (+ 2962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:54.019474 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.021872 (+ 2398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.022515 (+ 643us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:54.022811 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:54.022843 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 16:43:54.023001 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 16:43:54.023166 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6941,"mutex_wait_us":25,"thread_start_us":12650,"threads_started":1}
I20260926 16:43:54.024502 14435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 859f1dc2a56c4464b2ab26602702c5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:54.024556 14568 tablet_service.cc:1467] Processing CreateTablet for tablet 859f1dc2a56c4464b2ab26602702c5d4 (DEFAULT_TABLE table=test-table [id=4019732aa2bb482fba12a7de1390bab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:54.025935 14568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 859f1dc2a56c4464b2ab26602702c5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:54.028869 14701 tablet_service.cc:1467] Processing CreateTablet for tablet 859f1dc2a56c4464b2ab26602702c5d4 (DEFAULT_TABLE table=test-table [id=4019732aa2bb482fba12a7de1390bab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:54.030452 14701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 859f1dc2a56c4464b2ab26602702c5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:54.046124 14785 tablet_bootstrap.cc:492] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407: Bootstrap starting.
I20260926 16:43:54.050005 14785 tablet_bootstrap.cc:654] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:54.051244 14785 log.cc:824] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:54.054379 14786 tablet_bootstrap.cc:492] T 859f1dc2a56c4464b2ab26602702c5d4 P 439b87f2e3be4bd1ba1685f95d6dddc8: Bootstrap starting.
I20260926 16:43:54.056699 14787 tablet_bootstrap.cc:492] T 859f1dc2a56c4464b2ab26602702c5d4 P cbc1dd7520344f4c8e82af61ceb7a51e: Bootstrap starting.
I20260926 16:43:54.058352 14786 tablet_bootstrap.cc:654] T 859f1dc2a56c4464b2ab26602702c5d4 P 439b87f2e3be4bd1ba1685f95d6dddc8: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:54.059686 14786 log.cc:824] T 859f1dc2a56c4464b2ab26602702c5d4 P 439b87f2e3be4bd1ba1685f95d6dddc8: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:54.061729 14787 tablet_bootstrap.cc:654] T 859f1dc2a56c4464b2ab26602702c5d4 P cbc1dd7520344f4c8e82af61ceb7a51e: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:54.063499 14787 log.cc:824] T 859f1dc2a56c4464b2ab26602702c5d4 P cbc1dd7520344f4c8e82af61ceb7a51e: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:54.065033 14785 tablet_bootstrap.cc:492] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407: No bootstrap required, opened a new log
I20260926 16:43:54.065366 14785 ts_tablet_manager.cc:1397] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.000s
I20260926 16:43:54.067487 14786 tablet_bootstrap.cc:492] T 859f1dc2a56c4464b2ab26602702c5d4 P 439b87f2e3be4bd1ba1685f95d6dddc8: No bootstrap required, opened a new log
I20260926 16:43:54.067739 14787 tablet_bootstrap.cc:492] T 859f1dc2a56c4464b2ab26602702c5d4 P cbc1dd7520344f4c8e82af61ceb7a51e: No bootstrap required, opened a new log
I20260926 16:43:54.067975 14786 ts_tablet_manager.cc:1397] T 859f1dc2a56c4464b2ab26602702c5d4 P 439b87f2e3be4bd1ba1685f95d6dddc8: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20260926 16:43:54.068056 14787 ts_tablet_manager.cc:1397] T 859f1dc2a56c4464b2ab26602702c5d4 P cbc1dd7520344f4c8e82af61ceb7a51e: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260926 16:43:54.076247 14785 raft_consensus.cc:348] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a2302d8fb844f796ea23f2481a2407" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44653 } } peers { permanent_uuid: "439b87f2e3be4bd1ba1685f95d6dddc8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44589 } } peers { permanent_uuid: "cbc1dd7520344f4c8e82af61ceb7a51e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35983 } }
I20260926 16:43:54.076746 14785 raft_consensus.cc:374] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:54.076841 14785 raft_consensus.cc:729] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92a2302d8fb844f796ea23f2481a2407, State: Initialized, Role: FOLLOWER
I20260926 16:43:54.077652 14785 consensus_queue.cc:260] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a2302d8fb844f796ea23f2481a2407" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44653 } } peers { permanent_uuid: "439b87f2e3be4bd1ba1685f95d6dddc8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44589 } } peers { permanent_uuid: "cbc1dd7520344f4c8e82af61ceb7a51e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35983 } }
I20260926 16:43:54.078150 14786 raft_consensus.cc:348] T 859f1dc2a56c4464b2ab26602702c5d4 P 439b87f2e3be4bd1ba1685f95d6dddc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a2302d8fb844f796ea23f2481a2407" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44653 } } peers { permanent_uuid: "439b87f2e3be4bd1ba1685f95d6dddc8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44589 } } peers { permanent_uuid: "cbc1dd7520344f4c8e82af61ceb7a51e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35983 } }
I20260926 16:43:54.078785 14786 raft_consensus.cc:374] T 859f1dc2a56c4464b2ab26602702c5d4 P 439b87f2e3be4bd1ba1685f95d6dddc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:54.078924 14786 raft_consensus.cc:729] T 859f1dc2a56c4464b2ab26602702c5d4 P 439b87f2e3be4bd1ba1685f95d6dddc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 439b87f2e3be4bd1ba1685f95d6dddc8, State: Initialized, Role: FOLLOWER
I20260926 16:43:54.079105 14787 raft_consensus.cc:348] T 859f1dc2a56c4464b2ab26602702c5d4 P cbc1dd7520344f4c8e82af61ceb7a51e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a2302d8fb844f796ea23f2481a2407" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44653 } } peers { permanent_uuid: "439b87f2e3be4bd1ba1685f95d6dddc8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44589 } } peers { permanent_uuid: "cbc1dd7520344f4c8e82af61ceb7a51e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35983 } }
I20260926 16:43:54.079591 14787 raft_consensus.cc:374] T 859f1dc2a56c4464b2ab26602702c5d4 P cbc1dd7520344f4c8e82af61ceb7a51e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:54.079710 14787 raft_consensus.cc:729] T 859f1dc2a56c4464b2ab26602702c5d4 P cbc1dd7520344f4c8e82af61ceb7a51e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbc1dd7520344f4c8e82af61ceb7a51e, State: Initialized, Role: FOLLOWER
I20260926 16:43:54.079883 14633 heartbeater.cc:503] Master 127.9.238.254:44881 was elected leader, sending a full tablet report...
I20260926 16:43:54.080063 14786 consensus_queue.cc:260] T 859f1dc2a56c4464b2ab26602702c5d4 P 439b87f2e3be4bd1ba1685f95d6dddc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a2302d8fb844f796ea23f2481a2407" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44653 } } peers { permanent_uuid: "439b87f2e3be4bd1ba1685f95d6dddc8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44589 } } peers { permanent_uuid: "cbc1dd7520344f4c8e82af61ceb7a51e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35983 } }
I20260926 16:43:54.080482 14787 consensus_queue.cc:260] T 859f1dc2a56c4464b2ab26602702c5d4 P cbc1dd7520344f4c8e82af61ceb7a51e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a2302d8fb844f796ea23f2481a2407" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44653 } } peers { permanent_uuid: "439b87f2e3be4bd1ba1685f95d6dddc8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44589 } } peers { permanent_uuid: "cbc1dd7520344f4c8e82af61ceb7a51e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35983 } }
I20260926 16:43:54.081678 14785 ts_tablet_manager.cc:1428] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260926 16:43:54.083355 14766 heartbeater.cc:503] Master 127.9.238.254:44881 was elected leader, sending a full tablet report...
I20260926 16:43:54.083803 14787 ts_tablet_manager.cc:1428] T 859f1dc2a56c4464b2ab26602702c5d4 P cbc1dd7520344f4c8e82af61ceb7a51e: Time spent starting tablet: real 0.016s user 0.003s sys 0.013s
I20260926 16:43:54.087515 14786 ts_tablet_manager.cc:1428] T 859f1dc2a56c4464b2ab26602702c5d4 P 439b87f2e3be4bd1ba1685f95d6dddc8: Time spent starting tablet: real 0.019s user 0.012s sys 0.006s
W20260926 16:43:54.162279 14767 tablet.cc:2369] T 859f1dc2a56c4464b2ab26602702c5d4 P cbc1dd7520344f4c8e82af61ceb7a51e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:43:54.232698 14501 tablet.cc:2369] T 859f1dc2a56c4464b2ab26602702c5d4 P 439b87f2e3be4bd1ba1685f95d6dddc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:43:54.261499 14634 tablet.cc:2369] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:43:54.421969 14791 raft_consensus.cc:482] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:43:54.422405 14791 raft_consensus.cc:504] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a2302d8fb844f796ea23f2481a2407" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44653 } } peers { permanent_uuid: "439b87f2e3be4bd1ba1685f95d6dddc8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44589 } } peers { permanent_uuid: "cbc1dd7520344f4c8e82af61ceb7a51e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35983 } }
I20260926 16:43:54.424583 14791 leader_election.cc:288] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 439b87f2e3be4bd1ba1685f95d6dddc8 (127.9.238.193:44589), cbc1dd7520344f4c8e82af61ceb7a51e (127.9.238.195:35983)
I20260926 16:43:54.433620 14784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:54.425325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:49215 (local address 127.9.238.195:35983)
0926 16:43:54.425573 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:54.425584 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:54.426605 (+ 1021us) server_negotiation.cc:407] Connection header received
0926 16:43:54.426793 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:54.426799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:54.426890 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:54.427044 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:54.428035 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.428805 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.431428 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.432356 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.432579 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:54.432663 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:54.433154 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 16:43:54.433302 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 16:43:54.433650 14783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:54.425217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:59693 (local address 127.9.238.193:44589)
0926 16:43:54.425451 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:54.425458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:54.426356 (+ 898us) server_negotiation.cc:407] Connection header received
0926 16:43:54.426565 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:54.426573 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:54.426660 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:54.426784 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:43:54.427581 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.428746 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.431099 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.432279 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.432511 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:54.432612 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:54.433142 (+ 530us) server_negotiation.cc:299] Negotiation successful
0926 16:43:54.433308 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 16:43:54.434036 14794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:54.425238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:44589 (local address 127.9.238.194:59693)
0926 16:43:54.426204 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:54.426226 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:54.426390 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:54.426815 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:54.426820 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:54.426849 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:54.427361 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:54.427377 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.428913 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.428921 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:54.430912 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:54.430926 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.432572 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.432579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:54.432805 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:54.432984 (+ 179us) client_negotiation.cc:769] Sending connection context
0926 16:43:54.433111 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 16:43:54.433481 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":250,"thread_start_us":476,"threads_started":1}
I20260926 16:43:54.434463 14795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:54.425378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:35983 (local address 127.9.238.194:49215)
0926 16:43:54.426419 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:54.426444 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:54.426629 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:54.427120 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:54.427124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:54.427161 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:43:54.427801 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:54.427817 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.429031 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.429039 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:54.431136 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:54.431152 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.432635 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.432641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:54.432834 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:54.432940 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 16:43:54.433118 (+ 178us) client_negotiation.cc:240] Negotiation successful
0926 16:43:54.433473 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":892,"mutex_wait_us":123,"thread_start_us":551,"threads_started":1}
I20260926 16:43:54.435601 14721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "859f1dc2a56c4464b2ab26602702c5d4" candidate_uuid: "92a2302d8fb844f796ea23f2481a2407" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbc1dd7520344f4c8e82af61ceb7a51e" is_pre_election: true
I20260926 16:43:54.435608 14455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "859f1dc2a56c4464b2ab26602702c5d4" candidate_uuid: "92a2302d8fb844f796ea23f2481a2407" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "439b87f2e3be4bd1ba1685f95d6dddc8" is_pre_election: true
I20260926 16:43:54.436120 14721 raft_consensus.cc:2454] T 859f1dc2a56c4464b2ab26602702c5d4 P cbc1dd7520344f4c8e82af61ceb7a51e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92a2302d8fb844f796ea23f2481a2407 in term 0.
I20260926 16:43:54.436364 14455 raft_consensus.cc:2454] T 859f1dc2a56c4464b2ab26602702c5d4 P 439b87f2e3be4bd1ba1685f95d6dddc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92a2302d8fb844f796ea23f2481a2407 in term 0.
I20260926 16:43:54.436767 14524 leader_election.cc:302] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92a2302d8fb844f796ea23f2481a2407, cbc1dd7520344f4c8e82af61ceb7a51e; no voters:
I20260926 16:43:54.437273 14791 raft_consensus.cc:2781] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:43:54.437425 14791 raft_consensus.cc:482] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:43:54.437573 14791 raft_consensus.cc:3037] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:54.440366 14791 raft_consensus.cc:504] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a2302d8fb844f796ea23f2481a2407" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44653 } } peers { permanent_uuid: "439b87f2e3be4bd1ba1685f95d6dddc8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44589 } } peers { permanent_uuid: "cbc1dd7520344f4c8e82af61ceb7a51e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35983 } }
I20260926 16:43:54.441232 14791 leader_election.cc:288] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407 [CANDIDATE]: Term 1 election: Requested vote from peers 439b87f2e3be4bd1ba1685f95d6dddc8 (127.9.238.193:44589), cbc1dd7520344f4c8e82af61ceb7a51e (127.9.238.195:35983)
I20260926 16:43:54.441924 14455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "859f1dc2a56c4464b2ab26602702c5d4" candidate_uuid: "92a2302d8fb844f796ea23f2481a2407" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "439b87f2e3be4bd1ba1685f95d6dddc8"
I20260926 16:43:54.441988 14721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "859f1dc2a56c4464b2ab26602702c5d4" candidate_uuid: "92a2302d8fb844f796ea23f2481a2407" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbc1dd7520344f4c8e82af61ceb7a51e"
I20260926 16:43:54.442202 14455 raft_consensus.cc:3037] T 859f1dc2a56c4464b2ab26602702c5d4 P 439b87f2e3be4bd1ba1685f95d6dddc8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:54.442246 14721 raft_consensus.cc:3037] T 859f1dc2a56c4464b2ab26602702c5d4 P cbc1dd7520344f4c8e82af61ceb7a51e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:54.444731 14721 raft_consensus.cc:2454] T 859f1dc2a56c4464b2ab26602702c5d4 P cbc1dd7520344f4c8e82af61ceb7a51e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92a2302d8fb844f796ea23f2481a2407 in term 1.
I20260926 16:43:54.445127 14455 raft_consensus.cc:2454] T 859f1dc2a56c4464b2ab26602702c5d4 P 439b87f2e3be4bd1ba1685f95d6dddc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92a2302d8fb844f796ea23f2481a2407 in term 1.
I20260926 16:43:54.445348 14524 leader_election.cc:302] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92a2302d8fb844f796ea23f2481a2407, cbc1dd7520344f4c8e82af61ceb7a51e; no voters:
I20260926 16:43:54.445789 14791 raft_consensus.cc:2781] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:54.446406 14791 raft_consensus.cc:686] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407 [term 1 LEADER]: Becoming Leader. State: Replica: 92a2302d8fb844f796ea23f2481a2407, State: Running, Role: LEADER
I20260926 16:43:54.447261 14791 consensus_queue.cc:237] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a2302d8fb844f796ea23f2481a2407" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44653 } } peers { permanent_uuid: "439b87f2e3be4bd1ba1685f95d6dddc8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44589 } } peers { permanent_uuid: "cbc1dd7520344f4c8e82af61ceb7a51e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35983 } }
I20260926 16:43:54.453555 14311 catalog_manager.cc:5495] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92a2302d8fb844f796ea23f2481a2407 (127.9.238.194). New cstate: current_term: 1 leader_uuid: "92a2302d8fb844f796ea23f2481a2407" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92a2302d8fb844f796ea23f2481a2407" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "439b87f2e3be4bd1ba1685f95d6dddc8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbc1dd7520344f4c8e82af61ceb7a51e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35983 } health_report { overall_health: UNKNOWN } } }
I20260926 16:43:54.487046 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:54.480260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46002 (local address 127.9.238.194:44653)
0926 16:43:54.480571 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:54.480579 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:54.480801 (+ 222us) server_negotiation.cc:407] Connection header received
0926 16:43:54.481115 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:54.481120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:54.481225 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:54.481418 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:43:54.481911 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.482694 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.484210 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.484773 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.484994 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:54.485581 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:43:54.486277 (+ 696us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:43:54.486457 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:54.486594 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 16:43:54.486695 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260926 16:43:54.498147 14721 raft_consensus.cc:1261] T 859f1dc2a56c4464b2ab26602702c5d4 P cbc1dd7520344f4c8e82af61ceb7a51e [term 1 FOLLOWER]: Refusing update from remote peer 92a2302d8fb844f796ea23f2481a2407: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:43:54.498432 14455 raft_consensus.cc:1261] T 859f1dc2a56c4464b2ab26602702c5d4 P 439b87f2e3be4bd1ba1685f95d6dddc8 [term 1 FOLLOWER]: Refusing update from remote peer 92a2302d8fb844f796ea23f2481a2407: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:43:54.499609 14791 consensus_queue.cc:1035] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc1dd7520344f4c8e82af61ceb7a51e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:43:54.500080 14796 consensus_queue.cc:1035] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407 [LEADER]: Connected to new peer: Peer: permanent_uuid: "439b87f2e3be4bd1ba1685f95d6dddc8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:43:54.532527 14311 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:43:54.532900 14311 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:43:54.537147 14311 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 500363ef762e47edb00c1552be6f6f4b: Sending DeleteTablet for 3 replicas of tablet 859f1dc2a56c4464b2ab26602702c5d4
I20260926 16:43:54.538414 14701 tablet_service.cc:1514] Processing DeleteTablet for tablet 859f1dc2a56c4464b2ab26602702c5d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:54 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:57854
I20260926 16:43:54.538414 14568 tablet_service.cc:1514] Processing DeleteTablet for tablet 859f1dc2a56c4464b2ab26602702c5d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:54 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:45998
I20260926 16:43:54.538686 14435 tablet_service.cc:1514] Processing DeleteTablet for tablet 859f1dc2a56c4464b2ab26602702c5d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:54 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:53188
I20260926 16:43:54.541328 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 14373
I20260926 16:43:54.552503 14811 tablet_replica.cc:330] T 859f1dc2a56c4464b2ab26602702c5d4 P 439b87f2e3be4bd1ba1685f95d6dddc8: stopping tablet replica
I20260926 16:43:54.552850 14809 tablet_replica.cc:330] T 859f1dc2a56c4464b2ab26602702c5d4 P cbc1dd7520344f4c8e82af61ceb7a51e: stopping tablet replica
I20260926 16:43:54.553086 14811 raft_consensus.cc:2229] T 859f1dc2a56c4464b2ab26602702c5d4 P 439b87f2e3be4bd1ba1685f95d6dddc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:43:54.553429 14809 raft_consensus.cc:2229] T 859f1dc2a56c4464b2ab26602702c5d4 P cbc1dd7520344f4c8e82af61ceb7a51e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:43:54.553682 14811 raft_consensus.cc:2258] T 859f1dc2a56c4464b2ab26602702c5d4 P 439b87f2e3be4bd1ba1685f95d6dddc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:43:54.554054 14809 raft_consensus.cc:2258] T 859f1dc2a56c4464b2ab26602702c5d4 P cbc1dd7520344f4c8e82af61ceb7a51e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:43:54.554754 14810 tablet_replica.cc:330] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407: stopping tablet replica
I20260926 16:43:54.555272 14810 raft_consensus.cc:2229] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:43:54.556110 14810 raft_consensus.cc:2258] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:43:54.556444 14811 ts_tablet_manager.cc:1905] T 859f1dc2a56c4464b2ab26602702c5d4 P 439b87f2e3be4bd1ba1685f95d6dddc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:43:54.558092 14809 ts_tablet_manager.cc:1905] T 859f1dc2a56c4464b2ab26602702c5d4 P cbc1dd7520344f4c8e82af61ceb7a51e: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 16:43:54 dist-test-slave-wt2j krb5kdc[14268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.193@KRBTEST.COM
I20260926 16:43:54.564363 14811 ts_tablet_manager.cc:1918] T 859f1dc2a56c4464b2ab26602702c5d4 P 439b87f2e3be4bd1ba1685f95d6dddc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:43:54.564589 14811 log.cc:1196] T 859f1dc2a56c4464b2ab26602702c5d4 P 439b87f2e3be4bd1ba1685f95d6dddc8: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-0/wal/wals/859f1dc2a56c4464b2ab26602702c5d4
I20260926 16:43:54.565359 14811 ts_tablet_manager.cc:1939] T 859f1dc2a56c4464b2ab26602702c5d4 P 439b87f2e3be4bd1ba1685f95d6dddc8: Deleting consensus metadata
I20260926 16:43:54.566255 14810 ts_tablet_manager.cc:1905] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:43:54.567487 14297 catalog_manager.cc:4841] TS 439b87f2e3be4bd1ba1685f95d6dddc8 (127.9.238.193:44589): tablet 859f1dc2a56c4464b2ab26602702c5d4 (table test-table [id=4019732aa2bb482fba12a7de1390bab4]) successfully deleted
I20260926 16:43:54.570513 14783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:54.543138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.9.238.193:44589)
0926 16:43:54.543366 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:54.543373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:54.547565 (+ 4192us) server_negotiation.cc:407] Connection header received
0926 16:43:54.548974 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:54.548983 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:54.549108 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:54.549222 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:54.552208 (+ 2986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.553424 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.555928 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.556636 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.556737 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:54.562400 (+ 5663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:54.562414 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:54.562441 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:54.562498 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:54.566777 (+ 4279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:54.567841 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:54.567850 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:54.567854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:54.568055 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:54.568433 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:54.568440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:54.568447 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:54.568889 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:54.569202 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:54.569818 (+ 616us) server_negotiation.cc:299] Negotiation successful
0926 16:43:54.570054 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 16:43:54.571723 14809 ts_tablet_manager.cc:1918] T 859f1dc2a56c4464b2ab26602702c5d4 P cbc1dd7520344f4c8e82af61ceb7a51e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:43:54.571919 14809 log.cc:1196] T 859f1dc2a56c4464b2ab26602702c5d4 P cbc1dd7520344f4c8e82af61ceb7a51e: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-2/wal/wals/859f1dc2a56c4464b2ab26602702c5d4
I20260926 16:43:54.572074 14495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:43:54.572685 14809 ts_tablet_manager.cc:1939] T 859f1dc2a56c4464b2ab26602702c5d4 P cbc1dd7520344f4c8e82af61ceb7a51e: Deleting consensus metadata
I20260926 16:43:54.575256 14299 catalog_manager.cc:4841] TS cbc1dd7520344f4c8e82af61ceb7a51e (127.9.238.195:35983): tablet 859f1dc2a56c4464b2ab26602702c5d4 (table test-table [id=4019732aa2bb482fba12a7de1390bab4]) successfully deleted
I20260926 16:43:54.577920 14810 ts_tablet_manager.cc:1918] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:43:54.578131 14810 log.cc:1196] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790440988577176-10171-0/minicluster-data/ts-1/wal/wals/859f1dc2a56c4464b2ab26602702c5d4
I20260926 16:43:54.578848 14810 ts_tablet_manager.cc:1939] T 859f1dc2a56c4464b2ab26602702c5d4 P 92a2302d8fb844f796ea23f2481a2407: Deleting consensus metadata
I20260926 16:43:54.581161 14298 catalog_manager.cc:4841] TS 92a2302d8fb844f796ea23f2481a2407 (127.9.238.194:44653): tablet 859f1dc2a56c4464b2ab26602702c5d4 (table test-table [id=4019732aa2bb482fba12a7de1390bab4]) successfully deleted
I20260926 16:43:54.667200 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 14373
I20260926 16:43:54.695801 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 14506
Sep 26 16:43:54 dist-test-slave-wt2j krb5kdc[14268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.194@KRBTEST.COM
I20260926 16:43:54.715338 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:54.696926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46006 (local address 127.9.238.194:44653)
0926 16:43:54.697170 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:54.697177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:54.697207 (+ 30us) server_negotiation.cc:407] Connection header received
0926 16:43:54.697298 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:54.697304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:54.697407 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:54.697573 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:54.698998 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.700385 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.701818 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.702549 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.702698 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:54.707102 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:54.707126 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:54.707155 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:54.707226 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:54.711858 (+ 4632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:54.712800 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:54.712811 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:54.712815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:54.713020 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:54.713482 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:54.713487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:54.713491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:54.714031 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:54.714412 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:54.714596 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 16:43:54.714881 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 16:43:54.716427 14628 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:43:54.833200 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 14506
I20260926 16:43:54.853319 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 14639
Sep 26 16:43:54 dist-test-slave-wt2j krb5kdc[14268](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.195@KRBTEST.COM
I20260926 16:43:54.872027 14784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:54.854374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57858 (local address 127.9.238.195:35983)
0926 16:43:54.854639 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:54.854648 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:54.854779 (+ 131us) server_negotiation.cc:407] Connection header received
0926 16:43:54.855062 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:54.855067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:54.855147 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:54.855375 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:54.856991 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.857859 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.859258 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:54.859979 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:54.860191 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:54.865443 (+ 5252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:54.865464 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:54.865486 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:43:54.865552 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:54.868775 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:54.869578 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:54.869586 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:54.869589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:54.869755 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:43:54.870188 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:43:54.870203 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:43:54.870206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:43:54.870635 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:54.870967 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:54.871330 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 16:43:54.871605 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 16:43:54.872910 14761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:43:54.969456 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 14639
I20260926 16:43:54.986974 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 14281
I20260926 16:43:54.988487 14342 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:43:55.099779 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 14281
2026-09-26T16:43:55Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4934 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T16:43:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:43:55Z Disabled control of system clock
I20260926 16:43:55.178300 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:44325
--webserver_interface=127.9.238.254
--webserver_port=0
--builtin_ntp_servers=127.9.238.212:45103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:44325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 16:43:55.589107 14820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:55.589547 14820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:55.589645 14820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:55.598228 14820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:43:55.598408 14820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:55.598450 14820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:55.598510 14820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:43:55.598560 14820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:43:55.610797 14820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:45103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:44325
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:44325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:55.612610 14820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:55.614969 14820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:55.630119 14826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:55.630077 14828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:55.630180 14825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:55.633939 14820 server_base.cc:1034] running on GCE node
I20260926 16:43:55.635193 14820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:55.637496 14820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:55.638778 14820 hybrid_clock.cc:648] HybridClock initialized: now 1790441035638655 us; error 117 us; skew 500 ppm
I20260926 16:43:55.639271 14820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:43:55.642978 14820 webserver.cc:460] Webserver started at http://127.9.238.254:45713/ using document root <none> and password file <none>
I20260926 16:43:55.644073 14820 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:55.644243 14820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:55.644641 14820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:55.648535 14820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "a11935bc2fab431ea75bc93bca34f42b"
format_stamp: "Formatted at 2026-09-26 16:43:55 on dist-test-slave-wt2j"
server_key: "22554ec61fa6e523ba327b1b61ae76c2"
server_key_iv: "61fb636352872395b7fa7989a9ee7b05"
server_key_version: "encryptionkey@0"
I20260926 16:43:55.649557 14820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "a11935bc2fab431ea75bc93bca34f42b"
format_stamp: "Formatted at 2026-09-26 16:43:55 on dist-test-slave-wt2j"
server_key: "22554ec61fa6e523ba327b1b61ae76c2"
server_key_iv: "61fb636352872395b7fa7989a9ee7b05"
server_key_version: "encryptionkey@0"
I20260926 16:43:55.656435 14820 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 16:43:55.661291 14834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:55.663334 14820 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260926 16:43:55.663676 14820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "a11935bc2fab431ea75bc93bca34f42b"
format_stamp: "Formatted at 2026-09-26 16:43:55 on dist-test-slave-wt2j"
server_key: "22554ec61fa6e523ba327b1b61ae76c2"
server_key_iv: "61fb636352872395b7fa7989a9ee7b05"
server_key_version: "encryptionkey@0"
I20260926 16:43:55.663950 14820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:55.671239 14820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:55.672237 14820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:55.672533 14820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:55.695149 14820 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:44325
I20260926 16:43:55.695209 14885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:44325 every 8 connection(s)
I20260926 16:43:55.697036 14820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:43:55.701668 14886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:55.706027 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 14820
I20260926 16:43:55.706333 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:43:55.707049 10171 external_mini_cluster.cc:1428] Setting key 087f64ec358ccf09901851314b845ce8
I20260926 16:43:55.714610 14886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a11935bc2fab431ea75bc93bca34f42b: Bootstrap starting.
I20260926 16:43:55.717929 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:55.709370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37912 (local address 127.9.238.254:44325)
0926 16:43:55.710912 (+ 1542us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:55.710938 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:55.711008 (+ 70us) server_negotiation.cc:407] Connection header received
0926 16:43:55.713202 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:55.713235 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:55.713869 (+ 634us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:55.714388 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:55.715746 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:55.715752 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:55.715767 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:43:55.715781 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:55.715847 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:43:55.715863 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:55.716140 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:55.716312 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 16:43:55.716673 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":895,"thread_start_us":521,"threads_started":1}
I20260926 16:43:55.719156 14886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a11935bc2fab431ea75bc93bca34f42b: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:55.720460 14886 log.cc:824] T 00000000000000000000000000000000 P a11935bc2fab431ea75bc93bca34f42b: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:55.724186 14886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a11935bc2fab431ea75bc93bca34f42b: No bootstrap required, opened a new log
I20260926 16:43:55.731978 14886 raft_consensus.cc:348] T 00000000000000000000000000000000 P a11935bc2fab431ea75bc93bca34f42b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a11935bc2fab431ea75bc93bca34f42b" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44325 } }
I20260926 16:43:55.732360 14886 raft_consensus.cc:374] T 00000000000000000000000000000000 P a11935bc2fab431ea75bc93bca34f42b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:55.732487 14886 raft_consensus.cc:729] T 00000000000000000000000000000000 P a11935bc2fab431ea75bc93bca34f42b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a11935bc2fab431ea75bc93bca34f42b, State: Initialized, Role: FOLLOWER
I20260926 16:43:55.733171 14886 consensus_queue.cc:260] T 00000000000000000000000000000000 P a11935bc2fab431ea75bc93bca34f42b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a11935bc2fab431ea75bc93bca34f42b" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44325 } }
I20260926 16:43:55.733412 14886 raft_consensus.cc:388] T 00000000000000000000000000000000 P a11935bc2fab431ea75bc93bca34f42b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:43:55.733547 14886 raft_consensus.cc:482] T 00000000000000000000000000000000 P a11935bc2fab431ea75bc93bca34f42b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:43:55.733732 14886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a11935bc2fab431ea75bc93bca34f42b [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:55.736296 14886 raft_consensus.cc:504] T 00000000000000000000000000000000 P a11935bc2fab431ea75bc93bca34f42b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a11935bc2fab431ea75bc93bca34f42b" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44325 } }
I20260926 16:43:55.736915 14886 leader_election.cc:302] T 00000000000000000000000000000000 P a11935bc2fab431ea75bc93bca34f42b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a11935bc2fab431ea75bc93bca34f42b; no voters:
I20260926 16:43:55.737659 14886 leader_election.cc:288] T 00000000000000000000000000000000 P a11935bc2fab431ea75bc93bca34f42b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:43:55.737972 14891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a11935bc2fab431ea75bc93bca34f42b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:55.739251 14891 raft_consensus.cc:686] T 00000000000000000000000000000000 P a11935bc2fab431ea75bc93bca34f42b [term 1 LEADER]: Becoming Leader. State: Replica: a11935bc2fab431ea75bc93bca34f42b, State: Running, Role: LEADER
I20260926 16:43:55.740280 14886 sys_catalog.cc:564] T 00000000000000000000000000000000 P a11935bc2fab431ea75bc93bca34f42b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:43:55.740136 14891 consensus_queue.cc:237] T 00000000000000000000000000000000 P a11935bc2fab431ea75bc93bca34f42b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a11935bc2fab431ea75bc93bca34f42b" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44325 } }
I20260926 16:43:55.745589 14892 sys_catalog.cc:455] T 00000000000000000000000000000000 P a11935bc2fab431ea75bc93bca34f42b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a11935bc2fab431ea75bc93bca34f42b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a11935bc2fab431ea75bc93bca34f42b" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44325 } } }
I20260926 16:43:55.746167 14892 sys_catalog.cc:458] T 00000000000000000000000000000000 P a11935bc2fab431ea75bc93bca34f42b [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:55.746970 14893 sys_catalog.cc:455] T 00000000000000000000000000000000 P a11935bc2fab431ea75bc93bca34f42b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a11935bc2fab431ea75bc93bca34f42b. Latest consensus state: current_term: 1 leader_uuid: "a11935bc2fab431ea75bc93bca34f42b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a11935bc2fab431ea75bc93bca34f42b" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44325 } } }
I20260926 16:43:55.747709 14893 sys_catalog.cc:458] T 00000000000000000000000000000000 P a11935bc2fab431ea75bc93bca34f42b [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:55.751648 14899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:43:55.759624 14899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:43:55.773344 14899 catalog_manager.cc:1347] Generated new cluster ID: 2c857fbd9c2c4a8e86a08bf1cedcc2e3
I20260926 16:43:55.773510 14899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:43:55.786980 14899 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:43:55.787528 14899 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:43:55.801821 14899 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a11935bc2fab431ea75bc93bca34f42b: Generated new TSK 0
I20260926 16:43:55.803097 14899 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:43:55.813660 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:0
--local_ip_for_outbound_sockets=127.9.238.193
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44325
--builtin_ntp_servers=127.9.238.212:45103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 16:43:56.232370 14911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:56.232751 14911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:56.232852 14911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:56.241298 14911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:56.241509 14911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:56.241652 14911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:43:56.252847 14911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:45103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:56.254606 14911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:56.256928 14911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:56.273118 14917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:56.273353 14916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:56.273808 14919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:56.275271 14911 server_base.cc:1034] running on GCE node
I20260926 16:43:56.276276 14911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:56.277563 14911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:56.278813 14911 hybrid_clock.cc:648] HybridClock initialized: now 1790441036278728 us; error 69 us; skew 500 ppm
I20260926 16:43:56.279284 14911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:43:56.282377 14911 webserver.cc:460] Webserver started at http://127.9.238.193:44557/ using document root <none> and password file <none>
I20260926 16:43:56.283370 14911 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:56.283535 14911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:56.284037 14911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:56.287949 14911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-0/data/instance:
uuid: "f2026b23f5654f95a7c3483bb7603205"
format_stamp: "Formatted at 2026-09-26 16:43:56 on dist-test-slave-wt2j"
server_key: "5a7c253df1828c8bc6f96ed77a3459f2"
server_key_iv: "033ba2cdc8254986a76711bfd879f9ba"
server_key_version: "encryptionkey@0"
I20260926 16:43:56.289011 14911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance:
uuid: "f2026b23f5654f95a7c3483bb7603205"
format_stamp: "Formatted at 2026-09-26 16:43:56 on dist-test-slave-wt2j"
server_key: "5a7c253df1828c8bc6f96ed77a3459f2"
server_key_iv: "033ba2cdc8254986a76711bfd879f9ba"
server_key_version: "encryptionkey@0"
I20260926 16:43:56.296363 14911 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260926 16:43:56.301319 14925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:56.303257 14911 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 16:43:56.303515 14911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "f2026b23f5654f95a7c3483bb7603205"
format_stamp: "Formatted at 2026-09-26 16:43:56 on dist-test-slave-wt2j"
server_key: "5a7c253df1828c8bc6f96ed77a3459f2"
server_key_iv: "033ba2cdc8254986a76711bfd879f9ba"
server_key_version: "encryptionkey@0"
I20260926 16:43:56.303898 14911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:56.311635 14911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:56.312783 14911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:56.313143 14911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:56.314416 14911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:56.316321 14911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:56.316463 14911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:56.316629 14911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:56.316732 14911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:56.360283 14911 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:33483
I20260926 16:43:56.360363 15037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:33483 every 8 connection(s)
I20260926 16:43:56.362066 14911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:43:56.371657 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 14911
I20260926 16:43:56.371943 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance
I20260926 16:43:56.371984 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:56.365225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:57659 (local address 127.9.238.254:44325)
0926 16:43:56.365787 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:56.365791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:56.367448 (+ 1657us) server_negotiation.cc:407] Connection header received
0926 16:43:56.369527 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:56.369538 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:56.369595 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:56.369719 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:56.371168 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:56.371174 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:56.371179 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:43:56.371197 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:56.371247 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:43:56.371258 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:56.371401 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:56.371601 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 16:43:56.371673 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":295,"threads_started":1}
I20260926 16:43:56.372752 10171 external_mini_cluster.cc:1428] Setting key 70560f17dba8a6a1ecd344fd501e73d8
I20260926 16:43:56.373121 15041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:56.366104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:44325 (local address 127.9.238.193:57659)
0926 16:43:56.367251 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:56.367303 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:56.369123 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:56.369988 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:56.369997 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:56.370103 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:56.370837 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:43:56.370839 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:56.370882 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:43:56.370970 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:56.371353 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:56.371363 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:56.371372 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 16:43:56.371708 (+ 336us) client_negotiation.cc:240] Negotiation successful
0926 16:43:56.372005 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":334,"threads_started":1}
I20260926 16:43:56.374908 15038 heartbeater.cc:347] Connected to a master server at 127.9.238.254:44325
I20260926 16:43:56.375470 15038 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:56.376411 15038 heartbeater.cc:511] Master 127.9.238.254:44325 requested a full tablet report, sending...
I20260926 16:43:56.376825 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:0
--local_ip_for_outbound_sockets=127.9.238.194
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44325
--builtin_ntp_servers=127.9.238.212:45103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 16:43:56.379240 14850 ts_manager.cc:194] Registered new tserver with Master: f2026b23f5654f95a7c3483bb7603205 (127.9.238.193:33483)
W20260926 16:43:56.733349 15042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:56.733668 15042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:56.733810 15042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:56.741490 15042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:56.741639 15042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:56.741808 15042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:43:56.751415 15042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:45103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:56.752954 15042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:56.754781 15042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:56.769520 15047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:56.769683 15048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:56.770102 15050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:56.771795 15042 server_base.cc:1034] running on GCE node
I20260926 16:43:56.772697 15042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:56.774087 15042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:56.775391 15042 hybrid_clock.cc:648] HybridClock initialized: now 1790441036775334 us; error 50 us; skew 500 ppm
I20260926 16:43:56.775934 15042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:43:56.779290 15042 webserver.cc:460] Webserver started at http://127.9.238.194:39539/ using document root <none> and password file <none>
I20260926 16:43:56.780481 15042 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:56.780652 15042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:56.781092 15042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:56.785461 15042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-1/data/instance:
uuid: "0a6e32335908481ca6b8bf0fcc22e13e"
format_stamp: "Formatted at 2026-09-26 16:43:56 on dist-test-slave-wt2j"
server_key: "850f62f7b95d1b7318b7f163c91717f7"
server_key_iv: "3da934c1701f0e32beaf065931386c4a"
server_key_version: "encryptionkey@0"
I20260926 16:43:56.786475 15042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance:
uuid: "0a6e32335908481ca6b8bf0fcc22e13e"
format_stamp: "Formatted at 2026-09-26 16:43:56 on dist-test-slave-wt2j"
server_key: "850f62f7b95d1b7318b7f163c91717f7"
server_key_iv: "3da934c1701f0e32beaf065931386c4a"
server_key_version: "encryptionkey@0"
I20260926 16:43:56.793154 15042 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 16:43:56.797819 15056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:56.799786 15042 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 16:43:56.800047 15042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "0a6e32335908481ca6b8bf0fcc22e13e"
format_stamp: "Formatted at 2026-09-26 16:43:56 on dist-test-slave-wt2j"
server_key: "850f62f7b95d1b7318b7f163c91717f7"
server_key_iv: "3da934c1701f0e32beaf065931386c4a"
server_key_version: "encryptionkey@0"
I20260926 16:43:56.800352 15042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:56.807977 15042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:56.809165 15042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:56.809495 15042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:56.810663 15042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:56.812637 15042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:56.812780 15042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:56.812925 15042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:56.813017 15042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:56.854650 15042 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:43843
I20260926 16:43:56.854764 15168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:43843 every 8 connection(s)
I20260926 16:43:56.856416 15042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:43:56.863859 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 15042
I20260926 16:43:56.864094 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance
I20260926 16:43:56.864631 10171 external_mini_cluster.cc:1428] Setting key af2548dd93773159329ddb49e33d3ddd
I20260926 16:43:56.867408 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:56.859933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:50983 (local address 127.9.238.254:44325)
0926 16:43:56.860184 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:56.860189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:56.862285 (+ 2096us) server_negotiation.cc:407] Connection header received
0926 16:43:56.864263 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:56.864269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:56.864343 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:56.864466 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:56.866274 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:56.866281 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:56.866285 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:43:56.866310 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:56.866362 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:43:56.866379 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:56.866551 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:56.866879 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 16:43:56.866971 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 16:43:56.868500 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:56.860793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:44325 (local address 127.9.238.194:50983)
0926 16:43:56.862080 (+ 1287us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:56.862134 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:56.863908 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:56.864752 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:56.864767 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:56.864905 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:56.865691 (+ 786us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:43:56.865694 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:56.865735 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:43:56.865870 (+ 135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:56.866538 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:56.866548 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:56.866557 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 16:43:56.866878 (+ 321us) client_negotiation.cc:240] Negotiation successful
0926 16:43:56.867217 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":373,"threads_started":1}
I20260926 16:43:56.868860 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:0
--local_ip_for_outbound_sockets=127.9.238.195
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44325
--builtin_ntp_servers=127.9.238.212:45103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 16:43:56.870770 15169 heartbeater.cc:347] Connected to a master server at 127.9.238.254:44325
I20260926 16:43:56.871167 15169 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:56.871918 15169 heartbeater.cc:511] Master 127.9.238.254:44325 requested a full tablet report, sending...
I20260926 16:43:56.873725 14850 ts_manager.cc:194] Registered new tserver with Master: 0a6e32335908481ca6b8bf0fcc22e13e (127.9.238.194:43843)
W20260926 16:43:57.207991 15172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:57.208345 15172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:57.208544 15172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:57.216107 15172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:57.216220 15172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:57.216435 15172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:43:57.225812 15172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:45103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:57.227201 15172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:57.229089 15172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:57.243847 15177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:57.244086 15180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:57.244207 15178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:57.246049 15172 server_base.cc:1034] running on GCE node
I20260926 16:43:57.246973 15172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:57.248441 15172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:57.249745 15172 hybrid_clock.cc:648] HybridClock initialized: now 1790441037249662 us; error 71 us; skew 500 ppm
I20260926 16:43:57.250238 15172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:43:57.253750 15172 webserver.cc:460] Webserver started at http://127.9.238.195:33367/ using document root <none> and password file <none>
I20260926 16:43:57.254717 15172 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:57.254877 15172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:57.255276 15172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:57.258908 15172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-2/data/instance:
uuid: "597b1293c3c44901b90bb933388c83a4"
format_stamp: "Formatted at 2026-09-26 16:43:57 on dist-test-slave-wt2j"
server_key: "d3eb74f53d6e139388795343f57d0821"
server_key_iv: "1b060a1655ad401c033ff035a9c1a9ff"
server_key_version: "encryptionkey@0"
I20260926 16:43:57.259971 15172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance:
uuid: "597b1293c3c44901b90bb933388c83a4"
format_stamp: "Formatted at 2026-09-26 16:43:57 on dist-test-slave-wt2j"
server_key: "d3eb74f53d6e139388795343f57d0821"
server_key_iv: "1b060a1655ad401c033ff035a9c1a9ff"
server_key_version: "encryptionkey@0"
I20260926 16:43:57.267627 15172 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260926 16:43:57.272830 15186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:57.274719 15172 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 16:43:57.274998 15172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "597b1293c3c44901b90bb933388c83a4"
format_stamp: "Formatted at 2026-09-26 16:43:57 on dist-test-slave-wt2j"
server_key: "d3eb74f53d6e139388795343f57d0821"
server_key_iv: "1b060a1655ad401c033ff035a9c1a9ff"
server_key_version: "encryptionkey@0"
I20260926 16:43:57.275303 15172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:57.282678 15172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:57.283874 15172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:57.284201 15172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:57.285370 15172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:43:57.287170 15172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:43:57.287307 15172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:57.287459 15172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:43:57.287585 15172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:57.328065 15172 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:40141
I20260926 16:43:57.328176 15298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:40141 every 8 connection(s)
I20260926 16:43:57.329566 15172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:43:57.336148 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 15172
I20260926 16:43:57.336374 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance
I20260926 16:43:57.336910 10171 external_mini_cluster.cc:1428] Setting key f9c15edf174439b9a2537969df57220b
I20260926 16:43:57.340986 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:57.333371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:41237 (local address 127.9.238.254:44325)
0926 16:43:57.333643 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:57.333649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:57.335601 (+ 1952us) server_negotiation.cc:407] Connection header received
0926 16:43:57.337758 (+ 2157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:57.337766 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:57.337892 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:57.338074 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:57.339776 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:57.339782 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:57.339787 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:43:57.339808 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:57.339851 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:43:57.339864 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:57.340010 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:57.340382 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 16:43:57.340454 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260926 16:43:57.341912 15301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:57.333991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:44325 (local address 127.9.238.195:41237)
0926 16:43:57.335346 (+ 1355us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:57.335411 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:57.337327 (+ 1916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:57.338402 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:57.338418 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:57.338583 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:57.339383 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:43:57.339386 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:57.339427 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:43:57.339521 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:57.340020 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:57.340031 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:57.340050 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 16:43:57.340362 (+ 312us) client_negotiation.cc:240] Negotiation successful
0926 16:43:57.340696 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":405,"threads_started":1}
I20260926 16:43:57.344010 15299 heartbeater.cc:347] Connected to a master server at 127.9.238.254:44325
I20260926 16:43:57.344383 15299 heartbeater.cc:464] Registering TS with master...
I20260926 16:43:57.345073 15299 heartbeater.cc:511] Master 127.9.238.254:44325 requested a full tablet report, sending...
I20260926 16:43:57.346951 14850 ts_manager.cc:194] Registered new tserver with Master: 597b1293c3c44901b90bb933388c83a4 (127.9.238.195:40141)
I20260926 16:43:57.354615 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:43:57.364125 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:57.360760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37918 (local address 127.9.238.254:44325)
0926 16:43:57.361083 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:57.361090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:57.361640 (+ 550us) server_negotiation.cc:407] Connection header received
0926 16:43:57.361823 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:57.361831 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:57.361916 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:57.362083 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:57.363285 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:57.363292 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:57.363298 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:43:57.363319 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:57.363376 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:43:57.363393 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:57.363600 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:57.363725 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 16:43:57.363796 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 16:43:57.369760 14850 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:43:57.377650 14850 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:43:57.383363 15038 heartbeater.cc:503] Master 127.9.238.254:44325 was elected leader, sending a full tablet report...
I20260926 16:43:57.427927 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:57.395215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36864 (local address 127.9.238.193:33483)
0926 16:43:57.422929 (+ 27714us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:57.422938 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:57.422981 (+ 43us) server_negotiation.cc:407] Connection header received
0926 16:43:57.423074 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:57.423084 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:57.423408 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:57.423650 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:57.426669 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:57.426685 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:57.426709 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:43:57.426738 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:57.426827 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:43:57.426856 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:57.427059 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:57.427329 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 16:43:57.427454 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27472,"thread_start_us":27280,"threads_started":1}
I20260926 16:43:57.428177 15314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:57.394886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:33483 (local address 127.0.0.1:36864)
0926 16:43:57.420457 (+ 25571us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:57.420539 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:57.420864 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:57.423819 (+ 2955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:57.423835 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:57.424045 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:57.425464 (+ 1419us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:43:57.425469 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:57.425533 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:43:57.426433 (+ 900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:57.427113 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:57.427117 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:57.427121 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 16:43:57.427347 (+ 226us) client_negotiation.cc:240] Negotiation successful
0926 16:43:57.427416 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25076,"thread_start_us":30700,"threads_started":1}
I20260926 16:43:57.428308 15312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:57.395093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:40141 (local address 127.0.0.1:59006)
0926 16:43:57.421711 (+ 26618us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:57.421742 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:57.421856 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:57.424776 (+ 2920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:57.424782 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:57.424811 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:57.426067 (+ 1256us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:43:57.426071 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:57.426088 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:43:57.426206 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:57.426837 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:57.426852 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:57.426862 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 16:43:57.427045 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 16:43:57.427157 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26486,"thread_start_us":26270,"threads_started":1}
I20260926 16:43:57.429327 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:57.395837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59006 (local address 127.9.238.195:40141)
0926 16:43:57.422926 (+ 27089us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:57.422938 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:57.422981 (+ 43us) server_negotiation.cc:407] Connection header received
0926 16:43:57.423074 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:57.423084 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:57.423408 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:57.423650 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:57.426527 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:57.426538 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:57.426556 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:43:57.426577 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:57.426655 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:43:57.426679 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:57.428666 (+ 1987us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:57.428769 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 16:43:57.428897 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26858,"thread_start_us":29387,"threads_started":1}
I20260926 16:43:57.430648 15316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:57.396094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47260 (local address 127.9.238.194:43843)
0926 16:43:57.419141 (+ 23047us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:57.419158 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:57.426157 (+ 6999us) server_negotiation.cc:407] Connection header received
0926 16:43:57.426311 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:57.426320 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:57.426672 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:57.426862 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:57.429557 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:57.429570 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:57.429588 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:43:57.429611 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:57.429689 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:43:57.429721 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:57.429833 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:57.430166 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 16:43:57.430309 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22705,"thread_start_us":23109,"threads_started":1}
I20260926 16:43:57.431226 15311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:57.395419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:43843 (local address 127.0.0.1:47260)
0926 16:43:57.425952 (+ 30533us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:57.425971 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:57.426166 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:57.427964 (+ 1798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:57.427970 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:57.428001 (+ 31us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:57.429264 (+ 1263us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:43:57.429267 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:57.429289 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:43:57.429359 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:57.429924 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:57.429929 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:57.429933 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 16:43:57.430073 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 16:43:57.430128 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30357,"thread_start_us":26528,"threads_started":1}
I20260926 16:43:57.433247 14973 tablet_service.cc:1467] Processing CreateTablet for tablet c9dd538502fb4217896b2eff668e4dad (DEFAULT_TABLE table=test-table [id=a5d4236af42f4633a96e5799589ec99f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:57.433512 15234 tablet_service.cc:1467] Processing CreateTablet for tablet c9dd538502fb4217896b2eff668e4dad (DEFAULT_TABLE table=test-table [id=a5d4236af42f4633a96e5799589ec99f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:57.435061 14973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9dd538502fb4217896b2eff668e4dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:57.435220 15234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9dd538502fb4217896b2eff668e4dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:57.436004 15104 tablet_service.cc:1467] Processing CreateTablet for tablet c9dd538502fb4217896b2eff668e4dad (DEFAULT_TABLE table=test-table [id=a5d4236af42f4633a96e5799589ec99f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:43:57.437619 15104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9dd538502fb4217896b2eff668e4dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:57.449730 15317 tablet_bootstrap.cc:492] T c9dd538502fb4217896b2eff668e4dad P 597b1293c3c44901b90bb933388c83a4: Bootstrap starting.
I20260926 16:43:57.452746 15318 tablet_bootstrap.cc:492] T c9dd538502fb4217896b2eff668e4dad P f2026b23f5654f95a7c3483bb7603205: Bootstrap starting.
I20260926 16:43:57.454602 15319 tablet_bootstrap.cc:492] T c9dd538502fb4217896b2eff668e4dad P 0a6e32335908481ca6b8bf0fcc22e13e: Bootstrap starting.
I20260926 16:43:57.454872 15317 tablet_bootstrap.cc:654] T c9dd538502fb4217896b2eff668e4dad P 597b1293c3c44901b90bb933388c83a4: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:57.456521 15317 log.cc:824] T c9dd538502fb4217896b2eff668e4dad P 597b1293c3c44901b90bb933388c83a4: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:57.456912 15318 tablet_bootstrap.cc:654] T c9dd538502fb4217896b2eff668e4dad P f2026b23f5654f95a7c3483bb7603205: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:57.458428 15319 tablet_bootstrap.cc:654] T c9dd538502fb4217896b2eff668e4dad P 0a6e32335908481ca6b8bf0fcc22e13e: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:57.458496 15318 log.cc:824] T c9dd538502fb4217896b2eff668e4dad P f2026b23f5654f95a7c3483bb7603205: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:57.459892 15319 log.cc:824] T c9dd538502fb4217896b2eff668e4dad P 0a6e32335908481ca6b8bf0fcc22e13e: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:57.464383 15318 tablet_bootstrap.cc:492] T c9dd538502fb4217896b2eff668e4dad P f2026b23f5654f95a7c3483bb7603205: No bootstrap required, opened a new log
I20260926 16:43:57.464574 15319 tablet_bootstrap.cc:492] T c9dd538502fb4217896b2eff668e4dad P 0a6e32335908481ca6b8bf0fcc22e13e: No bootstrap required, opened a new log
I20260926 16:43:57.464766 15318 ts_tablet_manager.cc:1397] T c9dd538502fb4217896b2eff668e4dad P f2026b23f5654f95a7c3483bb7603205: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260926 16:43:57.464912 15319 ts_tablet_manager.cc:1397] T c9dd538502fb4217896b2eff668e4dad P 0a6e32335908481ca6b8bf0fcc22e13e: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.001s
I20260926 16:43:57.465786 15317 tablet_bootstrap.cc:492] T c9dd538502fb4217896b2eff668e4dad P 597b1293c3c44901b90bb933388c83a4: No bootstrap required, opened a new log
I20260926 16:43:57.466078 15317 ts_tablet_manager.cc:1397] T c9dd538502fb4217896b2eff668e4dad P 597b1293c3c44901b90bb933388c83a4: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s
I20260926 16:43:57.473582 15318 raft_consensus.cc:348] T c9dd538502fb4217896b2eff668e4dad P f2026b23f5654f95a7c3483bb7603205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2026b23f5654f95a7c3483bb7603205" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 33483 } } peers { permanent_uuid: "597b1293c3c44901b90bb933388c83a4" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40141 } } peers { permanent_uuid: "0a6e32335908481ca6b8bf0fcc22e13e" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 43843 } }
I20260926 16:43:57.474103 15318 raft_consensus.cc:374] T c9dd538502fb4217896b2eff668e4dad P f2026b23f5654f95a7c3483bb7603205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:57.474200 15318 raft_consensus.cc:729] T c9dd538502fb4217896b2eff668e4dad P f2026b23f5654f95a7c3483bb7603205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2026b23f5654f95a7c3483bb7603205, State: Initialized, Role: FOLLOWER
I20260926 16:43:57.474947 15318 consensus_queue.cc:260] T c9dd538502fb4217896b2eff668e4dad P f2026b23f5654f95a7c3483bb7603205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2026b23f5654f95a7c3483bb7603205" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 33483 } } peers { permanent_uuid: "597b1293c3c44901b90bb933388c83a4" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40141 } } peers { permanent_uuid: "0a6e32335908481ca6b8bf0fcc22e13e" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 43843 } }
I20260926 16:43:57.475921 15319 raft_consensus.cc:348] T c9dd538502fb4217896b2eff668e4dad P 0a6e32335908481ca6b8bf0fcc22e13e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2026b23f5654f95a7c3483bb7603205" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 33483 } } peers { permanent_uuid: "597b1293c3c44901b90bb933388c83a4" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40141 } } peers { permanent_uuid: "0a6e32335908481ca6b8bf0fcc22e13e" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 43843 } }
I20260926 16:43:57.476481 15319 raft_consensus.cc:374] T c9dd538502fb4217896b2eff668e4dad P 0a6e32335908481ca6b8bf0fcc22e13e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:57.476576 15319 raft_consensus.cc:729] T c9dd538502fb4217896b2eff668e4dad P 0a6e32335908481ca6b8bf0fcc22e13e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a6e32335908481ca6b8bf0fcc22e13e, State: Initialized, Role: FOLLOWER
I20260926 16:43:57.477169 15318 ts_tablet_manager.cc:1428] T c9dd538502fb4217896b2eff668e4dad P f2026b23f5654f95a7c3483bb7603205: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260926 16:43:57.477387 15319 consensus_queue.cc:260] T c9dd538502fb4217896b2eff668e4dad P 0a6e32335908481ca6b8bf0fcc22e13e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2026b23f5654f95a7c3483bb7603205" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 33483 } } peers { permanent_uuid: "597b1293c3c44901b90bb933388c83a4" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40141 } } peers { permanent_uuid: "0a6e32335908481ca6b8bf0fcc22e13e" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 43843 } }
I20260926 16:43:57.477424 15317 raft_consensus.cc:348] T c9dd538502fb4217896b2eff668e4dad P 597b1293c3c44901b90bb933388c83a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2026b23f5654f95a7c3483bb7603205" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 33483 } } peers { permanent_uuid: "597b1293c3c44901b90bb933388c83a4" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40141 } } peers { permanent_uuid: "0a6e32335908481ca6b8bf0fcc22e13e" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 43843 } }
I20260926 16:43:57.477813 15317 raft_consensus.cc:374] T c9dd538502fb4217896b2eff668e4dad P 597b1293c3c44901b90bb933388c83a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:57.477908 15317 raft_consensus.cc:729] T c9dd538502fb4217896b2eff668e4dad P 597b1293c3c44901b90bb933388c83a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 597b1293c3c44901b90bb933388c83a4, State: Initialized, Role: FOLLOWER
I20260926 16:43:57.478647 15317 consensus_queue.cc:260] T c9dd538502fb4217896b2eff668e4dad P 597b1293c3c44901b90bb933388c83a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2026b23f5654f95a7c3483bb7603205" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 33483 } } peers { permanent_uuid: "597b1293c3c44901b90bb933388c83a4" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40141 } } peers { permanent_uuid: "0a6e32335908481ca6b8bf0fcc22e13e" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 43843 } }
I20260926 16:43:57.481184 15169 heartbeater.cc:503] Master 127.9.238.254:44325 was elected leader, sending a full tablet report...
I20260926 16:43:57.482488 15299 heartbeater.cc:503] Master 127.9.238.254:44325 was elected leader, sending a full tablet report...
I20260926 16:43:57.483107 15319 ts_tablet_manager.cc:1428] T c9dd538502fb4217896b2eff668e4dad P 0a6e32335908481ca6b8bf0fcc22e13e: Time spent starting tablet: real 0.018s user 0.011s sys 0.006s
I20260926 16:43:57.482456 15317 ts_tablet_manager.cc:1428] T c9dd538502fb4217896b2eff668e4dad P 597b1293c3c44901b90bb933388c83a4: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
W20260926 16:43:57.581836 15300 tablet.cc:2369] T c9dd538502fb4217896b2eff668e4dad P 597b1293c3c44901b90bb933388c83a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:43:57.609256 15170 tablet.cc:2369] T c9dd538502fb4217896b2eff668e4dad P 0a6e32335908481ca6b8bf0fcc22e13e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:43:57.616358 15039 tablet.cc:2369] T c9dd538502fb4217896b2eff668e4dad P f2026b23f5654f95a7c3483bb7603205: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:43:57.714118 15323 raft_consensus.cc:482] T c9dd538502fb4217896b2eff668e4dad P f2026b23f5654f95a7c3483bb7603205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:43:57.714620 15323 raft_consensus.cc:504] T c9dd538502fb4217896b2eff668e4dad P f2026b23f5654f95a7c3483bb7603205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2026b23f5654f95a7c3483bb7603205" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 33483 } } peers { permanent_uuid: "597b1293c3c44901b90bb933388c83a4" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40141 } } peers { permanent_uuid: "0a6e32335908481ca6b8bf0fcc22e13e" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 43843 } }
I20260926 16:43:57.717609 15323 leader_election.cc:288] T c9dd538502fb4217896b2eff668e4dad P f2026b23f5654f95a7c3483bb7603205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 597b1293c3c44901b90bb933388c83a4 (127.9.238.195:40141), 0a6e32335908481ca6b8bf0fcc22e13e (127.9.238.194:43843)
I20260926 16:43:57.730283 15316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:57.718549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:48613 (local address 127.9.238.194:43843)
0926 16:43:57.718813 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:57.718820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:57.719904 (+ 1084us) server_negotiation.cc:407] Connection header received
0926 16:43:57.720045 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:57.720054 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:57.720142 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:57.720344 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:57.729235 (+ 8891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:57.729241 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:57.729244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:43:57.729269 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:57.729309 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:43:57.729324 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:57.729454 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:57.729755 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 16:43:57.729840 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260926 16:43:57.730419 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:57.718239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:37089 (local address 127.9.238.195:40141)
0926 16:43:57.718477 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:57.718483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:57.719398 (+ 915us) server_negotiation.cc:407] Connection header received
0926 16:43:57.719530 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:57.719537 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:57.719671 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:57.719907 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:57.729366 (+ 9459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:57.729373 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:57.729414 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:43:57.729452 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:57.729525 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:43:57.729553 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:57.729724 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:57.729919 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 16:43:57.730030 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260926 16:43:57.730743 15326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:57.717929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:40141 (local address 127.9.238.193:37089)
0926 16:43:57.719186 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:57.719212 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:57.719386 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:57.719963 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:57.719969 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:57.720005 (+ 36us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:57.728816 (+ 8811us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:43:57.728820 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:57.728846 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:43:57.728942 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:57.729737 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:57.729742 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:57.729748 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 16:43:57.730099 (+ 351us) client_negotiation.cc:240] Negotiation successful
0926 16:43:57.730186 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1059,"thread_start_us":766,"threads_started":1}
I20260926 16:43:57.731227 15327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:57.718298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:43843 (local address 127.9.238.193:48613)
0926 16:43:57.719739 (+ 1441us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:57.719765 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:57.719910 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:57.720507 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:57.720513 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:57.720543 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:43:57.728759 (+ 8216us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:43:57.728764 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:43:57.728792 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:43:57.728880 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:43:57.729470 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:43:57.729477 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:43:57.729484 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 16:43:57.729778 (+ 294us) client_negotiation.cc:240] Negotiation successful
0926 16:43:57.729891 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1288,"thread_start_us":734,"threads_started":1}
I20260926 16:43:57.732417 15254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9dd538502fb4217896b2eff668e4dad" candidate_uuid: "f2026b23f5654f95a7c3483bb7603205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597b1293c3c44901b90bb933388c83a4" is_pre_election: true
I20260926 16:43:57.732638 15124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9dd538502fb4217896b2eff668e4dad" candidate_uuid: "f2026b23f5654f95a7c3483bb7603205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a6e32335908481ca6b8bf0fcc22e13e" is_pre_election: true
I20260926 16:43:57.732978 15254 raft_consensus.cc:2454] T c9dd538502fb4217896b2eff668e4dad P 597b1293c3c44901b90bb933388c83a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2026b23f5654f95a7c3483bb7603205 in term 0.
I20260926 16:43:57.733381 15124 raft_consensus.cc:2454] T c9dd538502fb4217896b2eff668e4dad P 0a6e32335908481ca6b8bf0fcc22e13e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2026b23f5654f95a7c3483bb7603205 in term 0.
I20260926 16:43:57.733829 14927 leader_election.cc:302] T c9dd538502fb4217896b2eff668e4dad P f2026b23f5654f95a7c3483bb7603205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 597b1293c3c44901b90bb933388c83a4, f2026b23f5654f95a7c3483bb7603205; no voters:
I20260926 16:43:57.734540 15323 raft_consensus.cc:2781] T c9dd538502fb4217896b2eff668e4dad P f2026b23f5654f95a7c3483bb7603205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:43:57.734767 15323 raft_consensus.cc:482] T c9dd538502fb4217896b2eff668e4dad P f2026b23f5654f95a7c3483bb7603205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:43:57.734910 15323 raft_consensus.cc:3037] T c9dd538502fb4217896b2eff668e4dad P f2026b23f5654f95a7c3483bb7603205 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:57.737689 15323 raft_consensus.cc:504] T c9dd538502fb4217896b2eff668e4dad P f2026b23f5654f95a7c3483bb7603205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2026b23f5654f95a7c3483bb7603205" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 33483 } } peers { permanent_uuid: "597b1293c3c44901b90bb933388c83a4" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40141 } } peers { permanent_uuid: "0a6e32335908481ca6b8bf0fcc22e13e" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 43843 } }
I20260926 16:43:57.738520 15323 leader_election.cc:288] T c9dd538502fb4217896b2eff668e4dad P f2026b23f5654f95a7c3483bb7603205 [CANDIDATE]: Term 1 election: Requested vote from peers 597b1293c3c44901b90bb933388c83a4 (127.9.238.195:40141), 0a6e32335908481ca6b8bf0fcc22e13e (127.9.238.194:43843)
I20260926 16:43:57.739216 15254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9dd538502fb4217896b2eff668e4dad" candidate_uuid: "f2026b23f5654f95a7c3483bb7603205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "597b1293c3c44901b90bb933388c83a4"
I20260926 16:43:57.739249 15124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9dd538502fb4217896b2eff668e4dad" candidate_uuid: "f2026b23f5654f95a7c3483bb7603205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a6e32335908481ca6b8bf0fcc22e13e"
I20260926 16:43:57.739495 15254 raft_consensus.cc:3037] T c9dd538502fb4217896b2eff668e4dad P 597b1293c3c44901b90bb933388c83a4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:57.739598 15124 raft_consensus.cc:3037] T c9dd538502fb4217896b2eff668e4dad P 0a6e32335908481ca6b8bf0fcc22e13e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:57.742280 15254 raft_consensus.cc:2454] T c9dd538502fb4217896b2eff668e4dad P 597b1293c3c44901b90bb933388c83a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2026b23f5654f95a7c3483bb7603205 in term 1.
I20260926 16:43:57.742884 15124 raft_consensus.cc:2454] T c9dd538502fb4217896b2eff668e4dad P 0a6e32335908481ca6b8bf0fcc22e13e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2026b23f5654f95a7c3483bb7603205 in term 1.
I20260926 16:43:57.742870 14927 leader_election.cc:302] T c9dd538502fb4217896b2eff668e4dad P f2026b23f5654f95a7c3483bb7603205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 597b1293c3c44901b90bb933388c83a4, f2026b23f5654f95a7c3483bb7603205; no voters:
I20260926 16:43:57.743304 15323 raft_consensus.cc:2781] T c9dd538502fb4217896b2eff668e4dad P f2026b23f5654f95a7c3483bb7603205 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:57.744007 15323 raft_consensus.cc:686] T c9dd538502fb4217896b2eff668e4dad P f2026b23f5654f95a7c3483bb7603205 [term 1 LEADER]: Becoming Leader. State: Replica: f2026b23f5654f95a7c3483bb7603205, State: Running, Role: LEADER
I20260926 16:43:57.744614 15323 consensus_queue.cc:237] T c9dd538502fb4217896b2eff668e4dad P f2026b23f5654f95a7c3483bb7603205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2026b23f5654f95a7c3483bb7603205" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 33483 } } peers { permanent_uuid: "597b1293c3c44901b90bb933388c83a4" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40141 } } peers { permanent_uuid: "0a6e32335908481ca6b8bf0fcc22e13e" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 43843 } }
I20260926 16:43:57.749966 14849 catalog_manager.cc:5495] T c9dd538502fb4217896b2eff668e4dad P f2026b23f5654f95a7c3483bb7603205 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2026b23f5654f95a7c3483bb7603205 (127.9.238.193). New cstate: current_term: 1 leader_uuid: "f2026b23f5654f95a7c3483bb7603205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2026b23f5654f95a7c3483bb7603205" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 33483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "597b1293c3c44901b90bb933388c83a4" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a6e32335908481ca6b8bf0fcc22e13e" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 43843 } health_report { overall_health: UNKNOWN } } }
I20260926 16:43:57.780292 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:57.777451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36868 (local address 127.9.238.193:33483)
0926 16:43:57.777648 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:57.777653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:57.777864 (+ 211us) server_negotiation.cc:407] Connection header received
0926 16:43:57.778121 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:57.778125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:57.778213 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:57.778323 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:57.779425 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:57.779429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:57.779432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:43:57.779447 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:57.779482 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:43:57.779494 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:57.779707 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:57.779934 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 16:43:57.780087 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 16:43:57.790956 15254 raft_consensus.cc:1261] T c9dd538502fb4217896b2eff668e4dad P 597b1293c3c44901b90bb933388c83a4 [term 1 FOLLOWER]: Refusing update from remote peer f2026b23f5654f95a7c3483bb7603205: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:43:57.790931 15124 raft_consensus.cc:1261] T c9dd538502fb4217896b2eff668e4dad P 0a6e32335908481ca6b8bf0fcc22e13e [term 1 FOLLOWER]: Refusing update from remote peer f2026b23f5654f95a7c3483bb7603205: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:43:57.792325 15328 consensus_queue.cc:1035] T c9dd538502fb4217896b2eff668e4dad P f2026b23f5654f95a7c3483bb7603205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a6e32335908481ca6b8bf0fcc22e13e" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 43843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:43:57.792876 15323 consensus_queue.cc:1035] T c9dd538502fb4217896b2eff668e4dad P f2026b23f5654f95a7c3483bb7603205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "597b1293c3c44901b90bb933388c83a4" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:43:57.809406 15333 mvcc.cc:205] Tried to move back new op lower bound from 7333646490779049984 to 7333646490606632960. Current Snapshot: MvccSnapshot[applied={T|T < 7333646490779049984}]
I20260926 16:43:57.828647 14849 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:43:57.829051 14849 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:43:57.833765 14849 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a11935bc2fab431ea75bc93bca34f42b: Sending DeleteTablet for 3 replicas of tablet c9dd538502fb4217896b2eff668e4dad
I20260926 16:43:57.835094 14973 tablet_service.cc:1514] Processing DeleteTablet for tablet c9dd538502fb4217896b2eff668e4dad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:57 UTC) from {username='slave'} at 127.0.0.1:36864
I20260926 16:43:57.835155 15104 tablet_service.cc:1514] Processing DeleteTablet for tablet c9dd538502fb4217896b2eff668e4dad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:57 UTC) from {username='slave'} at 127.0.0.1:47260
I20260926 16:43:57.835381 15234 tablet_service.cc:1514] Processing DeleteTablet for tablet c9dd538502fb4217896b2eff668e4dad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:43:57 UTC) from {username='slave'} at 127.0.0.1:59006
I20260926 16:43:57.837297 15343 tablet_replica.cc:330] T c9dd538502fb4217896b2eff668e4dad P 597b1293c3c44901b90bb933388c83a4: stopping tablet replica
I20260926 16:43:57.837983 15343 raft_consensus.cc:2229] T c9dd538502fb4217896b2eff668e4dad P 597b1293c3c44901b90bb933388c83a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:43:57.838084 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 14911
I20260926 16:43:57.839358 15343 raft_consensus.cc:2258] T c9dd538502fb4217896b2eff668e4dad P 597b1293c3c44901b90bb933388c83a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:43:57.842453 15343 ts_tablet_manager.cc:1905] T c9dd538502fb4217896b2eff668e4dad P 597b1293c3c44901b90bb933388c83a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:43:57.848730 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:57.840902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36876 (local address 127.9.238.193:33483)
0926 16:43:57.841155 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:57.841164 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:57.844087 (+ 2923us) server_negotiation.cc:407] Connection header received
0926 16:43:57.845031 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:57.845046 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:57.845136 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:57.845243 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:57.847285 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:57.847294 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:57.847302 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:43:57.847330 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:57.847394 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:43:57.847421 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:57.847625 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:57.848147 (+ 522us) server_negotiation.cc:299] Negotiation successful
0926 16:43:57.848240 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 16:43:57.850366 15033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:43:57.852962 15343 ts_tablet_manager.cc:1918] T c9dd538502fb4217896b2eff668e4dad P 597b1293c3c44901b90bb933388c83a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:43:57.853475 15343 log.cc:1196] T c9dd538502fb4217896b2eff668e4dad P 597b1293c3c44901b90bb933388c83a4: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-2/wal/wals/c9dd538502fb4217896b2eff668e4dad
I20260926 16:43:57.854206 15343 ts_tablet_manager.cc:1939] T c9dd538502fb4217896b2eff668e4dad P 597b1293c3c44901b90bb933388c83a4: Deleting consensus metadata
I20260926 16:43:57.855870 15341 tablet_replica.cc:330] T c9dd538502fb4217896b2eff668e4dad P 0a6e32335908481ca6b8bf0fcc22e13e: stopping tablet replica
I20260926 16:43:57.856631 15341 raft_consensus.cc:2229] T c9dd538502fb4217896b2eff668e4dad P 0a6e32335908481ca6b8bf0fcc22e13e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:43:57.857038 14836 catalog_manager.cc:4841] TS 597b1293c3c44901b90bb933388c83a4 (127.9.238.195:40141): tablet c9dd538502fb4217896b2eff668e4dad (table test-table [id=a5d4236af42f4633a96e5799589ec99f]) successfully deleted
I20260926 16:43:57.857405 15341 raft_consensus.cc:2258] T c9dd538502fb4217896b2eff668e4dad P 0a6e32335908481ca6b8bf0fcc22e13e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:43:57.861099 15341 ts_tablet_manager.cc:1905] T c9dd538502fb4217896b2eff668e4dad P 0a6e32335908481ca6b8bf0fcc22e13e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:43:57.871536 15341 ts_tablet_manager.cc:1918] T c9dd538502fb4217896b2eff668e4dad P 0a6e32335908481ca6b8bf0fcc22e13e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:43:57.871937 15341 log.cc:1196] T c9dd538502fb4217896b2eff668e4dad P 0a6e32335908481ca6b8bf0fcc22e13e: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790440988577176-10171-0/minicluster-data/ts-1/wal/wals/c9dd538502fb4217896b2eff668e4dad
I20260926 16:43:57.872613 15341 ts_tablet_manager.cc:1939] T c9dd538502fb4217896b2eff668e4dad P 0a6e32335908481ca6b8bf0fcc22e13e: Deleting consensus metadata
I20260926 16:43:57.875240 14837 catalog_manager.cc:4841] TS 0a6e32335908481ca6b8bf0fcc22e13e (127.9.238.194:43843): tablet c9dd538502fb4217896b2eff668e4dad (table test-table [id=a5d4236af42f4633a96e5799589ec99f]) successfully deleted
I20260926 16:43:57.970682 15342 tablet_replica.cc:330] T c9dd538502fb4217896b2eff668e4dad P f2026b23f5654f95a7c3483bb7603205: stopping tablet replica
I20260926 16:43:57.970996 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 14911
W20260926 16:43:57.992458 14838 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.238.193:33483 (error 108)
I20260926 16:43:57.992769 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 15042
I20260926 16:43:57.995142 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:57.994280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:33483 (local address 127.0.0.1:36892)
0926 16:43:57.994789 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:57.994894 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:33483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":285,"threads_started":1}
W20260926 16:43:57.995450 14838 catalog_manager.cc:4584] TS f2026b23f5654f95a7c3483bb7603205 (127.9.238.193:33483): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c9dd538502fb4217896b2eff668e4dad: Network error: Client connection negotiation failed: client connection to 127.9.238.193:33483: connect: Connection refused (error 111)
I20260926 16:43:57.997000 15316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:57.993766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47264 (local address 127.9.238.194:43843)
0926 16:43:57.993972 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:57.993980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:57.993999 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:43:57.994270 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:57.994276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:57.994374 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:57.994570 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:57.996301 (+ 1731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:57.996306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:57.996310 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:43:57.996329 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:57.996367 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:43:57.996381 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:57.996541 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:57.996620 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 16:43:57.996692 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 16:43:57.997714 15164 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:43:58.049587 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:58.048880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:33483 (local address 127.0.0.1:36898)
0926 16:43:58.049198 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:58.049381 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:33483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260926 16:43:58.050797 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:58.050368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:33483 (local address 127.0.0.1:36908)
0926 16:43:58.050542 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:58.050639 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:33483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 16:43:58.098732 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 15042
I20260926 16:43:58.112128 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:58.111365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:33483 (local address 127.0.0.1:36924)
0926 16:43:58.111752 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:58.111910 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:33483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260926 16:43:58.114331 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:58.113613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:33483 (local address 127.0.0.1:36936)
0926 16:43:58.113859 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:58.114017 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:33483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 16:43:58.117802 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 15172
I20260926 16:43:58.120774 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:58.118595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59018 (local address 127.9.238.195:40141)
0926 16:43:58.118774 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:58.118779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:58.118800 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:43:58.118978 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:58.118982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:58.119083 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:58.119195 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:58.120210 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:58.120215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:58.120218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:43:58.120232 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:58.120266 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:43:58.120279 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:58.120392 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:58.120486 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 16:43:58.120535 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 16:43:58.121667 15294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:43:58.195413 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:58.194682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:33483 (local address 127.0.0.1:36938)
0926 16:43:58.195019 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:58.195165 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:33483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260926 16:43:58.196867 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:58.196309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:33483 (local address 127.0.0.1:36942)
0926 16:43:58.196584 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:58.196673 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:33483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260926 16:43:58.214290 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 15172
I20260926 16:43:58.233578 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 14820
I20260926 16:43:58.235142 14881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:43:58.344238 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 14820
2026-09-26T16:43:58Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3243 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 16:43:58.389020 10171 mini_oidc.cc:152] Starting JWKS server
I20260926 16:43:58.392315 10171 webserver.cc:460] Webserver started at https://127.0.0.1:46027/ using document root <none> and password file <none>
I20260926 16:43:58.392704 10171 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 16:43:58.393543 10171 webserver.cc:460] Webserver started at http://127.0.0.1:37239/ using document root <none> and password file <none>
2026-09-26T16:43:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:43:58Z Disabled control of system clock
I20260926 16:43:58.428156 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:44449
--webserver_interface=127.9.238.254
--webserver_port=0
--builtin_ntp_servers=127.9.238.212:34799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:44449
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46027/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440988577176-10171-0/testchainca.cert with env {}
W20260926 16:43:58.768492 15356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:58.768870 15356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:58.769035 15356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:58.776857 15356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:43:58.777107 15356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:58.777196 15356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:58.777328 15356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:43:58.777426 15356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 16:43:58.777529 15356 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 16:43:58.777643 15356 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46027/jwks.json
I20260926 16:43:58.788828 15356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:34799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:44449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:44449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46027/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440988577176-10171-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:58.790195 15356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:58.791574 15356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:58.804807 15361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:58.804850 15364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:58.805095 15362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:58.806871 15356 server_base.cc:1034] running on GCE node
I20260926 16:43:58.808168 15356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:58.810071 15356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:58.811407 15356 hybrid_clock.cc:648] HybridClock initialized: now 1790441038811361 us; error 40 us; skew 500 ppm
I20260926 16:43:58.811898 15356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:43:58.815049 15356 webserver.cc:460] Webserver started at http://127.9.238.254:37909/ using document root <none> and password file <none>
I20260926 16:43:58.816129 15356 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:58.816301 15356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:58.816697 15356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:58.820711 15356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "ac78c9d0d6e5432595170c21b6f5e9e0"
format_stamp: "Formatted at 2026-09-26 16:43:58 on dist-test-slave-wt2j"
server_key: "431b52dc1a0f8876a99e34dffba0b7ba"
server_key_iv: "57dadbaa40a5cf91dc3a935558160e79"
server_key_version: "encryptionkey@0"
I20260926 16:43:58.821856 15356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "ac78c9d0d6e5432595170c21b6f5e9e0"
format_stamp: "Formatted at 2026-09-26 16:43:58 on dist-test-slave-wt2j"
server_key: "431b52dc1a0f8876a99e34dffba0b7ba"
server_key_iv: "57dadbaa40a5cf91dc3a935558160e79"
server_key_version: "encryptionkey@0"
I20260926 16:43:58.829249 15356 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.006s
I20260926 16:43:58.834594 15370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:58.836711 15356 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 16:43:58.837009 15356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "ac78c9d0d6e5432595170c21b6f5e9e0"
format_stamp: "Formatted at 2026-09-26 16:43:58 on dist-test-slave-wt2j"
server_key: "431b52dc1a0f8876a99e34dffba0b7ba"
server_key_iv: "57dadbaa40a5cf91dc3a935558160e79"
server_key_version: "encryptionkey@0"
I20260926 16:43:58.837333 15356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:58.861007 15356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:58.862180 15356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:58.862603 15356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:58.886312 15356 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:44449
I20260926 16:43:58.886386 15421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:44449 every 8 connection(s)
I20260926 16:43:58.888159 15356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:43:58.892951 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 15356
I20260926 16:43:58.893018 15422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:58.893325 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:43:58.894052 10171 external_mini_cluster.cc:1428] Setting key 693178f63025a25c83b41ef5d18a9d90
I20260926 16:43:58.908834 15422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac78c9d0d6e5432595170c21b6f5e9e0: Bootstrap starting.
I20260926 16:43:58.911914 15425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:58.896409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44840 (local address 127.9.238.254:44449)
0926 16:43:58.897724 (+ 1315us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:58.897738 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:58.897794 (+ 56us) server_negotiation.cc:407] Connection header received
0926 16:43:58.899977 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:58.900006 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:58.900645 (+ 639us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:58.901141 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:58.901945 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:58.903439 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:58.905685 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:58.906712 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:58.906932 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:58.908560 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:58.908569 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:58.908590 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:43:58.909400 (+ 810us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:58.909508 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:43:58.909533 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:58.909642 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:58.909928 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 16:43:58.910505 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":473,"threads_started":1}
I20260926 16:43:58.913097 15422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac78c9d0d6e5432595170c21b6f5e9e0: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:58.914405 15422 log.cc:824] T 00000000000000000000000000000000 P ac78c9d0d6e5432595170c21b6f5e9e0: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:58.917320 10171 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:43:58.918325 15422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac78c9d0d6e5432595170c21b6f5e9e0: No bootstrap required, opened a new log
I20260926 16:43:58.927031 15422 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac78c9d0d6e5432595170c21b6f5e9e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac78c9d0d6e5432595170c21b6f5e9e0" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44449 } }
I20260926 16:43:58.927423 15422 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac78c9d0d6e5432595170c21b6f5e9e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:58.927486 15422 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac78c9d0d6e5432595170c21b6f5e9e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac78c9d0d6e5432595170c21b6f5e9e0, State: Initialized, Role: FOLLOWER
I20260926 16:43:58.928120 15422 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac78c9d0d6e5432595170c21b6f5e9e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac78c9d0d6e5432595170c21b6f5e9e0" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44449 } }
I20260926 16:43:58.928329 15422 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac78c9d0d6e5432595170c21b6f5e9e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:43:58.928422 15422 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac78c9d0d6e5432595170c21b6f5e9e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:43:58.928560 15422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac78c9d0d6e5432595170c21b6f5e9e0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:58.930721 15422 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac78c9d0d6e5432595170c21b6f5e9e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac78c9d0d6e5432595170c21b6f5e9e0" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44449 } }
I20260926 16:43:58.931205 15422 leader_election.cc:302] T 00000000000000000000000000000000 P ac78c9d0d6e5432595170c21b6f5e9e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac78c9d0d6e5432595170c21b6f5e9e0; no voters:
I20260926 16:43:58.931984 15422 leader_election.cc:288] T 00000000000000000000000000000000 P ac78c9d0d6e5432595170c21b6f5e9e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:43:58.932229 15427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac78c9d0d6e5432595170c21b6f5e9e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:58.933540 15427 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac78c9d0d6e5432595170c21b6f5e9e0 [term 1 LEADER]: Becoming Leader. State: Replica: ac78c9d0d6e5432595170c21b6f5e9e0, State: Running, Role: LEADER
I20260926 16:43:58.934273 15427 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac78c9d0d6e5432595170c21b6f5e9e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac78c9d0d6e5432595170c21b6f5e9e0" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44449 } }
I20260926 16:43:58.934530 15422 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac78c9d0d6e5432595170c21b6f5e9e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:43:58.938150 15429 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac78c9d0d6e5432595170c21b6f5e9e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac78c9d0d6e5432595170c21b6f5e9e0. Latest consensus state: current_term: 1 leader_uuid: "ac78c9d0d6e5432595170c21b6f5e9e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac78c9d0d6e5432595170c21b6f5e9e0" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44449 } } }
I20260926 16:43:58.938182 15428 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac78c9d0d6e5432595170c21b6f5e9e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac78c9d0d6e5432595170c21b6f5e9e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac78c9d0d6e5432595170c21b6f5e9e0" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44449 } } }
I20260926 16:43:58.938781 15429 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac78c9d0d6e5432595170c21b6f5e9e0 [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:58.938781 15428 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac78c9d0d6e5432595170c21b6f5e9e0 [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:58.939954 15436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:43:58.951666 15436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:43:58.962883 15436 catalog_manager.cc:1347] Generated new cluster ID: c160928e2ade42a4918d2d49fae16c5c
I20260926 16:43:58.963052 15436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:43:58.980202 15436 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:43:58.981729 15436 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:43:58.992061 15436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac78c9d0d6e5432595170c21b6f5e9e0: Generated new TSK 0
I20260926 16:43:58.993419 15436 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 16:43:59.053838 15452 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 16:43:59.054069 15452 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 16:43:59.055053 15425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:59.022893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44850 (local address 127.9.238.254:44449)
0926 16:43:59.023174 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:59.023181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:59.023459 (+ 278us) server_negotiation.cc:407] Connection header received
0926 16:43:59.024016 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:59.024024 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:59.024146 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:59.024371 (+ 225us) server_negotiation.cc:226] Negotiated authn=JWT
0926 16:43:59.025157 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:59.026295 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:59.028813 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:59.029693 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:59.029818 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:59.030781 (+ 963us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 16:43:59.054250 (+ 23469us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 16:43:59.054695 (+ 445us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44850: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":99}
W20260926 16:43:59.055235 15425 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44850: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 16:43:59.056828 15451 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:43:59.022659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:44449 (local address 127.0.0.1:44850)
0926 16:43:59.023327 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:59.023351 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:59.023623 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:59.024478 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:59.024487 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:59.024504 (+ 17us) client_negotiation.cc:189] Negotiated authn=JWT
0926 16:43:59.024903 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:59.024917 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:59.026454 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:59.026459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:59.028589 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:59.028608 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:59.030113 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:59.030121 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:59.030354 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:59.030454 (+ 100us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 16:43:59.054650 (+ 24196us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 16:43:59.055081 (+ 431us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.238.254:44449: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":311,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.9.238.254:44449: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46027/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 16:43:59.059469 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 15356
I20260926 16:43:59.060878 15417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:43:59.170807 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 15356
2026-09-26T16:43:59Z chronyd exiting
I20260926 16:43:59.224310 10171 test_util.cc:175] -----------------------------------------------
I20260926 16:43:59.224542 10171 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790440988577176-10171-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (840 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 16:43:59.229835 10171 mini_oidc.cc:152] Starting JWKS server
I20260926 16:43:59.232386 10171 webserver.cc:460] Webserver started at https://127.0.0.1:34647/ using document root <none> and password file <none>
I20260926 16:43:59.232661 10171 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 16:43:59.233471 10171 webserver.cc:460] Webserver started at http://127.0.0.1:41415/ using document root <none> and password file <none>
2026-09-26T16:43:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:43:59Z Disabled control of system clock
I20260926 16:43:59.269582 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:34911
--webserver_interface=127.9.238.254
--webserver_port=0
--builtin_ntp_servers=127.9.238.212:38199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:34911
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34647/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440988577176-10171-0/testchainca.cert with env {}
W20260926 16:43:59.606231 15459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:43:59.606633 15459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:43:59.606810 15459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:43:59.614442 15459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:43:59.614640 15459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:43:59.614714 15459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:43:59.614830 15459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:43:59.614919 15459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 16:43:59.615011 15459 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 16:43:59.615082 15459 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34647/jwks.json
I20260926 16:43:59.625766 15459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:38199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:34911
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:34911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34647/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440988577176-10171-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:43:59.627295 15459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:43:59.628713 15459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:43:59.643890 15467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:59.644112 15465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:43:59.643898 15464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:43:59.645100 15459 server_base.cc:1034] running on GCE node
I20260926 16:43:59.646047 15459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:43:59.647867 15459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:43:59.649148 15459 hybrid_clock.cc:648] HybridClock initialized: now 1790441039649084 us; error 57 us; skew 500 ppm
I20260926 16:43:59.649590 15459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:43:59.652666 15459 webserver.cc:460] Webserver started at http://127.9.238.254:36393/ using document root <none> and password file <none>
I20260926 16:43:59.653512 15459 fs_manager.cc:359] Metadata directory not provided
I20260926 16:43:59.653662 15459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:43:59.654017 15459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:43:59.657260 15459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "c84e07a80be5478687d64e2dbfd70660"
format_stamp: "Formatted at 2026-09-26 16:43:59 on dist-test-slave-wt2j"
server_key: "f801c326a13a5223f9feaf241997ed7c"
server_key_iv: "e87902aedf908f2d77975b9234a29bcc"
server_key_version: "encryptionkey@0"
I20260926 16:43:59.658136 15459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "c84e07a80be5478687d64e2dbfd70660"
format_stamp: "Formatted at 2026-09-26 16:43:59 on dist-test-slave-wt2j"
server_key: "f801c326a13a5223f9feaf241997ed7c"
server_key_iv: "e87902aedf908f2d77975b9234a29bcc"
server_key_version: "encryptionkey@0"
I20260926 16:43:59.664290 15459 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 16:43:59.668730 15473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:43:59.670706 15459 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:43:59.670948 15459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "c84e07a80be5478687d64e2dbfd70660"
format_stamp: "Formatted at 2026-09-26 16:43:59 on dist-test-slave-wt2j"
server_key: "f801c326a13a5223f9feaf241997ed7c"
server_key_iv: "e87902aedf908f2d77975b9234a29bcc"
server_key_version: "encryptionkey@0"
I20260926 16:43:59.671197 15459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:43:59.683779 15459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:43:59.684722 15459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:43:59.685065 15459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:43:59.707171 15459 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:34911
I20260926 16:43:59.707190 15524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:34911 every 8 connection(s)
I20260926 16:43:59.708666 15459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:43:59.713706 15525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:43:59.715718 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 15459
I20260926 16:43:59.715920 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:43:59.716826 10171 external_mini_cluster.cc:1428] Setting key d22be90c8b107809d3d4850e33bdc756
I20260926 16:43:59.730175 15525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c84e07a80be5478687d64e2dbfd70660: Bootstrap starting.
I20260926 16:43:59.736083 15525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c84e07a80be5478687d64e2dbfd70660: Neither blocks nor log segments found. Creating new log.
I20260926 16:43:59.736996 15528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:59.719149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51576 (local address 127.9.238.254:34911)
0926 16:43:59.720551 (+ 1402us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:59.720570 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:59.720647 (+ 77us) server_negotiation.cc:407] Connection header received
0926 16:43:59.723179 (+ 2532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:59.723218 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:59.724074 (+ 856us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:59.724650 (+ 576us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:43:59.725391 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:59.727685 (+ 2294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:59.729272 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:59.730456 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:59.730682 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:59.732159 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:43:59.732173 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:43:59.732201 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:43:59.733154 (+ 953us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:43:59.733302 (+ 148us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:43:59.733332 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:43:59.733467 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 16:43:59.733794 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 16:43:59.734208 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":463,"threads_started":1}
I20260926 16:43:59.738389 15525 log.cc:824] T 00000000000000000000000000000000 P c84e07a80be5478687d64e2dbfd70660: Log is configured to *not* fsync() on all Append() calls
I20260926 16:43:59.742462 15525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c84e07a80be5478687d64e2dbfd70660: No bootstrap required, opened a new log
I20260926 16:43:59.742466 10171 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:43:59.751912 15525 raft_consensus.cc:348] T 00000000000000000000000000000000 P c84e07a80be5478687d64e2dbfd70660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84e07a80be5478687d64e2dbfd70660" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 34911 } }
I20260926 16:43:59.752466 15525 raft_consensus.cc:374] T 00000000000000000000000000000000 P c84e07a80be5478687d64e2dbfd70660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:43:59.752583 15525 raft_consensus.cc:729] T 00000000000000000000000000000000 P c84e07a80be5478687d64e2dbfd70660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84e07a80be5478687d64e2dbfd70660, State: Initialized, Role: FOLLOWER
I20260926 16:43:59.753400 15525 consensus_queue.cc:260] T 00000000000000000000000000000000 P c84e07a80be5478687d64e2dbfd70660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84e07a80be5478687d64e2dbfd70660" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 34911 } }
I20260926 16:43:59.753714 15525 raft_consensus.cc:388] T 00000000000000000000000000000000 P c84e07a80be5478687d64e2dbfd70660 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:43:59.753881 15525 raft_consensus.cc:482] T 00000000000000000000000000000000 P c84e07a80be5478687d64e2dbfd70660 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:43:59.754101 15525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c84e07a80be5478687d64e2dbfd70660 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:43:59.757336 15525 raft_consensus.cc:504] T 00000000000000000000000000000000 P c84e07a80be5478687d64e2dbfd70660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84e07a80be5478687d64e2dbfd70660" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 34911 } }
I20260926 16:43:59.758083 15525 leader_election.cc:302] T 00000000000000000000000000000000 P c84e07a80be5478687d64e2dbfd70660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c84e07a80be5478687d64e2dbfd70660; no voters:
I20260926 16:43:59.758841 15525 leader_election.cc:288] T 00000000000000000000000000000000 P c84e07a80be5478687d64e2dbfd70660 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:43:59.759079 15530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c84e07a80be5478687d64e2dbfd70660 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:43:59.760104 15530 raft_consensus.cc:686] T 00000000000000000000000000000000 P c84e07a80be5478687d64e2dbfd70660 [term 1 LEADER]: Becoming Leader. State: Replica: c84e07a80be5478687d64e2dbfd70660, State: Running, Role: LEADER
I20260926 16:43:59.760782 15530 consensus_queue.cc:237] T 00000000000000000000000000000000 P c84e07a80be5478687d64e2dbfd70660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84e07a80be5478687d64e2dbfd70660" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 34911 } }
I20260926 16:43:59.761591 15525 sys_catalog.cc:564] T 00000000000000000000000000000000 P c84e07a80be5478687d64e2dbfd70660 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:43:59.764165 15532 sys_catalog.cc:455] T 00000000000000000000000000000000 P c84e07a80be5478687d64e2dbfd70660 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c84e07a80be5478687d64e2dbfd70660. Latest consensus state: current_term: 1 leader_uuid: "c84e07a80be5478687d64e2dbfd70660" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84e07a80be5478687d64e2dbfd70660" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 34911 } } }
I20260926 16:43:59.764751 15532 sys_catalog.cc:458] T 00000000000000000000000000000000 P c84e07a80be5478687d64e2dbfd70660 [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:59.764765 15531 sys_catalog.cc:455] T 00000000000000000000000000000000 P c84e07a80be5478687d64e2dbfd70660 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c84e07a80be5478687d64e2dbfd70660" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84e07a80be5478687d64e2dbfd70660" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 34911 } } }
I20260926 16:43:59.765308 15531 sys_catalog.cc:458] T 00000000000000000000000000000000 P c84e07a80be5478687d64e2dbfd70660 [sys.catalog]: This master's current role is: LEADER
I20260926 16:43:59.766781 15537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:43:59.772680 15537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:43:59.784010 15537 catalog_manager.cc:1347] Generated new cluster ID: 43fd9f60cbd949d19174d5d96e485ace
I20260926 16:43:59.784193 15537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:43:59.827515 15537 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:43:59.829485 15537 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:43:59.847218 15537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c84e07a80be5478687d64e2dbfd70660: Generated new TSK 0
I20260926 16:43:59.848531 15537 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 16:43:59.894021 15555 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 16:43:59.894227 15555 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 16:43:59.895592 15528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:43:59.870844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51584 (local address 127.9.238.254:34911)
0926 16:43:59.871053 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 16:43:59.871065 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:43:59.871624 (+ 559us) server_negotiation.cc:407] Connection header received
0926 16:43:59.871980 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:43:59.871985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:43:59.872079 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:43:59.872294 (+ 215us) server_negotiation.cc:226] Negotiated authn=JWT
0926 16:43:59.872937 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:59.873863 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:59.876100 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:43:59.876958 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:43:59.877101 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:59.877895 (+ 794us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 16:43:59.894736 (+ 16841us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34647/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 16:43:59.895165 (+ 429us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51584: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34647/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 16:43:59.895747 15528 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51584: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34647/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 16:43:59.896129 15554 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:43:59.870585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:34911 (local address 127.0.0.1:51584)
0926 16:43:59.871410 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0926 16:43:59.871433 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:43:59.871669 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:43:59.872323 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:43:59.872329 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:43:59.872342 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0926 16:43:59.872726 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:59.872742 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:59.874044 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:59.874057 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:59.875857 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:43:59.875874 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:43:59.877176 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:43:59.877184 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:43:59.877408 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:43:59.877464 (+ 56us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 16:43:59.895108 (+ 17644us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34647/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 16:43:59.895496 (+ 388us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.238.254:34911: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34647/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":454,"threads_started":1}
I20260926 16:43:59.898242 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 15459
I20260926 16:43:59.900324 15520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:00.014194 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 15459
2026-09-26T16:44:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (843 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 16:44:00.072700 10171 mini_oidc.cc:152] Starting JWKS server
I20260926 16:44:00.074865 10171 webserver.cc:460] Webserver started at https://127.0.0.1:34067/ using document root <none> and password file <none>
I20260926 16:44:00.075189 10171 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 16:44:00.076018 10171 webserver.cc:460] Webserver started at http://127.0.0.1:34541/ using document root <none> and password file <none>
2026-09-26T16:44:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:44:00Z Disabled control of system clock
I20260926 16:44:00.112130 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:36927
--webserver_interface=127.9.238.254
--webserver_port=0
--builtin_ntp_servers=127.9.238.212:41029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:36927
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34067/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 16:44:00.446869 15562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:00.447263 15562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:00.447432 15562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:00.454895 15562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:44:00.455082 15562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:00.455168 15562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:00.455279 15562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:44:00.455358 15562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 16:44:00.455447 15562 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 16:44:00.455528 15562 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34067/jwks.json
I20260926 16:44:00.465205 15562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:41029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:36927
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:36927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34067/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:00.466638 15562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:00.468478 15562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:00.481354 15570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:00.481376 15567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:00.481376 15568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:00.482645 15562 server_base.cc:1034] running on GCE node
I20260926 16:44:00.483857 15562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:00.485803 15562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:00.487109 15562 hybrid_clock.cc:648] HybridClock initialized: now 1790441040487046 us; error 46 us; skew 500 ppm
I20260926 16:44:00.487741 15562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:44:00.491300 15562 webserver.cc:460] Webserver started at http://127.9.238.254:37149/ using document root <none> and password file <none>
I20260926 16:44:00.492401 15562 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:00.492575 15562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:00.492991 15562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:00.496892 15562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "fc8bd6bcd5fc40b9ab488a936845a105"
format_stamp: "Formatted at 2026-09-26 16:44:00 on dist-test-slave-wt2j"
server_key: "6241bbea810f45d0077d725a2947cce5"
server_key_iv: "fd694cf587fb44241f4f1f34a7a3ee92"
server_key_version: "encryptionkey@0"
I20260926 16:44:00.497936 15562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "fc8bd6bcd5fc40b9ab488a936845a105"
format_stamp: "Formatted at 2026-09-26 16:44:00 on dist-test-slave-wt2j"
server_key: "6241bbea810f45d0077d725a2947cce5"
server_key_iv: "fd694cf587fb44241f4f1f34a7a3ee92"
server_key_version: "encryptionkey@0"
I20260926 16:44:00.505704 15562 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260926 16:44:00.510684 15576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:00.513131 15562 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260926 16:44:00.513446 15562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "fc8bd6bcd5fc40b9ab488a936845a105"
format_stamp: "Formatted at 2026-09-26 16:44:00 on dist-test-slave-wt2j"
server_key: "6241bbea810f45d0077d725a2947cce5"
server_key_iv: "fd694cf587fb44241f4f1f34a7a3ee92"
server_key_version: "encryptionkey@0"
I20260926 16:44:00.513772 15562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:00.532450 15562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:00.533566 15562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:00.533934 15562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:00.557736 15562 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:36927
I20260926 16:44:00.557775 15627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:36927 every 8 connection(s)
I20260926 16:44:00.559695 15562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:44:00.564293 15628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:00.569806 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 15562
I20260926 16:44:00.570048 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:44:00.570776 10171 external_mini_cluster.cc:1428] Setting key 486b91c0ab256ffa2d575870036de6cf
I20260926 16:44:00.577121 15628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc8bd6bcd5fc40b9ab488a936845a105: Bootstrap starting.
I20260926 16:44:00.581146 15628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc8bd6bcd5fc40b9ab488a936845a105: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:00.582593 15628 log.cc:824] T 00000000000000000000000000000000 P fc8bd6bcd5fc40b9ab488a936845a105: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:00.585980 15628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc8bd6bcd5fc40b9ab488a936845a105: No bootstrap required, opened a new log
I20260926 16:44:00.589306 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:00.573424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42604 (local address 127.9.238.254:36927)
0926 16:44:00.574928 (+ 1504us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:00.574943 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:00.575001 (+ 58us) server_negotiation.cc:407] Connection header received
0926 16:44:00.577140 (+ 2139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:00.577173 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:00.577917 (+ 744us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:00.578393 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:00.579184 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:00.580781 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:00.582932 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:00.584072 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:00.584340 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:00.585999 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:00.586010 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:00.586030 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:44:00.586854 (+ 824us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:00.586959 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:44:00.586986 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:00.587172 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:00.587466 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 16:44:00.587943 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":878,"thread_start_us":562,"threads_started":1}
I20260926 16:44:00.594955 10171 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:44:00.595412 15628 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc8bd6bcd5fc40b9ab488a936845a105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8bd6bcd5fc40b9ab488a936845a105" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 36927 } }
I20260926 16:44:00.595889 15628 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc8bd6bcd5fc40b9ab488a936845a105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:00.595995 15628 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc8bd6bcd5fc40b9ab488a936845a105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc8bd6bcd5fc40b9ab488a936845a105, State: Initialized, Role: FOLLOWER
I20260926 16:44:00.596587 15628 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc8bd6bcd5fc40b9ab488a936845a105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8bd6bcd5fc40b9ab488a936845a105" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 36927 } }
I20260926 16:44:00.596834 15628 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc8bd6bcd5fc40b9ab488a936845a105 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:44:00.596974 15628 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc8bd6bcd5fc40b9ab488a936845a105 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:44:00.597165 15628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc8bd6bcd5fc40b9ab488a936845a105 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:00.599779 15628 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc8bd6bcd5fc40b9ab488a936845a105 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8bd6bcd5fc40b9ab488a936845a105" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 36927 } }
I20260926 16:44:00.600322 15628 leader_election.cc:302] T 00000000000000000000000000000000 P fc8bd6bcd5fc40b9ab488a936845a105 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc8bd6bcd5fc40b9ab488a936845a105; no voters:
I20260926 16:44:00.601015 15628 leader_election.cc:288] T 00000000000000000000000000000000 P fc8bd6bcd5fc40b9ab488a936845a105 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:44:00.601313 15633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc8bd6bcd5fc40b9ab488a936845a105 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:44:00.602358 15633 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc8bd6bcd5fc40b9ab488a936845a105 [term 1 LEADER]: Becoming Leader. State: Replica: fc8bd6bcd5fc40b9ab488a936845a105, State: Running, Role: LEADER
I20260926 16:44:00.603180 15633 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc8bd6bcd5fc40b9ab488a936845a105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8bd6bcd5fc40b9ab488a936845a105" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 36927 } }
I20260926 16:44:00.603569 15628 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc8bd6bcd5fc40b9ab488a936845a105 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:44:00.607870 15635 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc8bd6bcd5fc40b9ab488a936845a105 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc8bd6bcd5fc40b9ab488a936845a105. Latest consensus state: current_term: 1 leader_uuid: "fc8bd6bcd5fc40b9ab488a936845a105" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8bd6bcd5fc40b9ab488a936845a105" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 36927 } } }
I20260926 16:44:00.608350 15635 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc8bd6bcd5fc40b9ab488a936845a105 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:00.609072 15634 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc8bd6bcd5fc40b9ab488a936845a105 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc8bd6bcd5fc40b9ab488a936845a105" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc8bd6bcd5fc40b9ab488a936845a105" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 36927 } } }
I20260926 16:44:00.609468 15634 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc8bd6bcd5fc40b9ab488a936845a105 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:00.611055 15640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:44:00.618901 15640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:44:00.631214 15640 catalog_manager.cc:1347] Generated new cluster ID: 29cd7b69572c4694ac5948af460f3d2b
I20260926 16:44:00.631420 15640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:44:00.666392 15640 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:44:00.668026 15640 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:44:00.679795 15640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc8bd6bcd5fc40b9ab488a936845a105: Generated new TSK 0
I20260926 16:44:00.681435 15640 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:44:00.723477 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:00.713706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42610 (local address 127.9.238.254:36927)
0926 16:44:00.713926 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:00.713931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:00.714243 (+ 312us) server_negotiation.cc:407] Connection header received
0926 16:44:00.714542 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:00.714549 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:00.714670 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:00.714934 (+ 264us) server_negotiation.cc:226] Negotiated authn=JWT
0926 16:44:00.715496 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:00.716822 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:00.718336 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:00.719123 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:00.719272 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:00.720052 (+ 780us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 16:44:00.722739 (+ 2687us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34067/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 16:44:00.723120 (+ 381us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42610: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34067/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 16:44:00.723630 15631 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42610: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34067/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 16:44:00.725929 15657 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:00.713461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:36927 (local address 127.0.0.1:42610)
0926 16:44:00.714110 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:00.714128 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:00.714274 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:00.714943 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:00.714946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:00.714957 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0926 16:44:00.715299 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:00.715308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:00.717024 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:00.717030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:00.718178 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:00.718189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:00.719360 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:00.719365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:00.719499 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:00.719539 (+ 40us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 16:44:00.723062 (+ 3523us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34067/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 16:44:00.725358 (+ 2296us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.238.254:36927: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34067/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":1735,"thread_start_us":362,"threads_started":1}
I20260926 16:44:00.727175 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 15562
I20260926 16:44:00.728554 15623 generic_service.cc:196] Checking for leaks (request via RPC)
E20260926 16:44:00.735841 15658 webserver.cc:533] Webserver: sslize failed:
E20260926 16:44:00.735947 15658 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 16:44:00.837467 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 15562
2026-09-26T16:44:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (822 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T16:44:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:44:00Z Disabled control of system clock
I20260926 16:44:00.926580 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:46263
--webserver_interface=127.9.238.254
--webserver_port=0
--builtin_ntp_servers=127.9.238.212:43083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:46263
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440988577176-10171-0/testchainca.cert with env {}
W20260926 16:44:01.273941 15663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:01.274284 15663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:01.274376 15663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:01.281625 15663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:44:01.281757 15663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:01.281795 15663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:01.281847 15663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:44:01.281901 15663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 16:44:01.281958 15663 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 16:44:01.282011 15663 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 16:44:01.292186 15663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:43083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:46263
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:46263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440988577176-10171-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:01.293609 15663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:01.294957 15663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:01.308730 15669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:01.308759 15671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:01.308707 15668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:01.309954 15663 server_base.cc:1034] running on GCE node
I20260926 16:44:01.310920 15663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:01.312668 15663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:01.313935 15663 hybrid_clock.cc:648] HybridClock initialized: now 1790441041313890 us; error 56 us; skew 500 ppm
I20260926 16:44:01.314363 15663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:44:01.317621 15663 webserver.cc:460] Webserver started at http://127.9.238.254:42449/ using document root <none> and password file <none>
I20260926 16:44:01.318415 15663 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:01.318562 15663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:01.318889 15663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:01.322113 15663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "116f2c2b0b4a4e018cdeae5b99e57665"
format_stamp: "Formatted at 2026-09-26 16:44:01 on dist-test-slave-wt2j"
server_key: "b5499070e687bc477fcf3ced687b57a8"
server_key_iv: "49267807200275987bcca0b26f8f1ef6"
server_key_version: "encryptionkey@0"
I20260926 16:44:01.323334 15663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "116f2c2b0b4a4e018cdeae5b99e57665"
format_stamp: "Formatted at 2026-09-26 16:44:01 on dist-test-slave-wt2j"
server_key: "b5499070e687bc477fcf3ced687b57a8"
server_key_iv: "49267807200275987bcca0b26f8f1ef6"
server_key_version: "encryptionkey@0"
I20260926 16:44:01.329972 15663 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 16:44:01.334250 15677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:01.336160 15663 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 16:44:01.336395 15663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "116f2c2b0b4a4e018cdeae5b99e57665"
format_stamp: "Formatted at 2026-09-26 16:44:01 on dist-test-slave-wt2j"
server_key: "b5499070e687bc477fcf3ced687b57a8"
server_key_iv: "49267807200275987bcca0b26f8f1ef6"
server_key_version: "encryptionkey@0"
I20260926 16:44:01.336655 15663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:01.359365 15663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:01.360321 15663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:01.360635 15663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:01.381856 15663 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:46263
I20260926 16:44:01.381876 15728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:46263 every 8 connection(s)
I20260926 16:44:01.383364 15663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:44:01.384429 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 15663
I20260926 16:44:01.384716 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:44:01.385566 10171 external_mini_cluster.cc:1428] Setting key 9f63ba5accad966d55e516c742517d82
I20260926 16:44:01.387823 15729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:01.403127 15729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 116f2c2b0b4a4e018cdeae5b99e57665: Bootstrap starting.
I20260926 16:44:01.404723 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:01.388171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36058 (local address 127.9.238.254:46263)
0926 16:44:01.389495 (+ 1324us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:01.389508 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:01.389567 (+ 59us) server_negotiation.cc:407] Connection header received
0926 16:44:01.392591 (+ 3024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:01.392624 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:01.393361 (+ 737us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:01.393888 (+ 527us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:01.394632 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:01.396487 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:01.398446 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:01.399425 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:01.399700 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:01.401175 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:01.401184 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:01.401205 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:44:01.402168 (+ 963us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:01.402361 (+ 193us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:44:01.402391 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:01.402506 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:01.402827 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 16:44:01.403380 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":485,"threads_started":1}
I20260926 16:44:01.407184 15729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 116f2c2b0b4a4e018cdeae5b99e57665: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:01.408643 15729 log.cc:824] T 00000000000000000000000000000000 P 116f2c2b0b4a4e018cdeae5b99e57665: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:01.410192 10171 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:44:01.412825 15729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 116f2c2b0b4a4e018cdeae5b99e57665: No bootstrap required, opened a new log
I20260926 16:44:01.421155 15729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 116f2c2b0b4a4e018cdeae5b99e57665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116f2c2b0b4a4e018cdeae5b99e57665" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46263 } }
I20260926 16:44:01.421571 15729 raft_consensus.cc:374] T 00000000000000000000000000000000 P 116f2c2b0b4a4e018cdeae5b99e57665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:01.421653 15729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 116f2c2b0b4a4e018cdeae5b99e57665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 116f2c2b0b4a4e018cdeae5b99e57665, State: Initialized, Role: FOLLOWER
I20260926 16:44:01.422259 15729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 116f2c2b0b4a4e018cdeae5b99e57665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116f2c2b0b4a4e018cdeae5b99e57665" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46263 } }
I20260926 16:44:01.422477 15729 raft_consensus.cc:388] T 00000000000000000000000000000000 P 116f2c2b0b4a4e018cdeae5b99e57665 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:44:01.422592 15729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 116f2c2b0b4a4e018cdeae5b99e57665 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:44:01.422725 15729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 116f2c2b0b4a4e018cdeae5b99e57665 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:01.425477 15729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 116f2c2b0b4a4e018cdeae5b99e57665 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116f2c2b0b4a4e018cdeae5b99e57665" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46263 } }
I20260926 16:44:01.426044 15729 leader_election.cc:302] T 00000000000000000000000000000000 P 116f2c2b0b4a4e018cdeae5b99e57665 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 116f2c2b0b4a4e018cdeae5b99e57665; no voters:
I20260926 16:44:01.426723 15729 leader_election.cc:288] T 00000000000000000000000000000000 P 116f2c2b0b4a4e018cdeae5b99e57665 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:44:01.426997 15734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 116f2c2b0b4a4e018cdeae5b99e57665 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:44:01.428104 15734 raft_consensus.cc:686] T 00000000000000000000000000000000 P 116f2c2b0b4a4e018cdeae5b99e57665 [term 1 LEADER]: Becoming Leader. State: Replica: 116f2c2b0b4a4e018cdeae5b99e57665, State: Running, Role: LEADER
I20260926 16:44:01.428843 15734 consensus_queue.cc:237] T 00000000000000000000000000000000 P 116f2c2b0b4a4e018cdeae5b99e57665 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116f2c2b0b4a4e018cdeae5b99e57665" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46263 } }
I20260926 16:44:01.429548 15729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 116f2c2b0b4a4e018cdeae5b99e57665 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:44:01.432333 15735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 116f2c2b0b4a4e018cdeae5b99e57665 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "116f2c2b0b4a4e018cdeae5b99e57665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116f2c2b0b4a4e018cdeae5b99e57665" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46263 } } }
I20260926 16:44:01.432284 15736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 116f2c2b0b4a4e018cdeae5b99e57665 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 116f2c2b0b4a4e018cdeae5b99e57665. Latest consensus state: current_term: 1 leader_uuid: "116f2c2b0b4a4e018cdeae5b99e57665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "116f2c2b0b4a4e018cdeae5b99e57665" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46263 } } }
I20260926 16:44:01.432847 15735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 116f2c2b0b4a4e018cdeae5b99e57665 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:01.432847 15736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 116f2c2b0b4a4e018cdeae5b99e57665 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:01.436348 15739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:44:01.443465 15739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:44:01.456051 15739 catalog_manager.cc:1347] Generated new cluster ID: d190baccb35c470986a4aebd497e4d53
I20260926 16:44:01.456312 15739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:44:01.470989 15739 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:44:01.472909 15739 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:44:01.484145 15739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 116f2c2b0b4a4e018cdeae5b99e57665: Generated new TSK 0
I20260926 16:44:01.485488 15739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:44:01.686748 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:01.507864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36068 (local address 127.9.238.254:46263)
0926 16:44:01.508180 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:01.508202 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:01.508655 (+ 453us) server_negotiation.cc:407] Connection header received
0926 16:44:01.508990 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:01.508995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:01.509104 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:01.509406 (+ 302us) server_negotiation.cc:226] Negotiated authn=JWT
0926 16:44:01.510095 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:01.511059 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:01.513544 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:01.514300 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:01.514448 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:01.515273 (+ 825us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 16:44:01.685779 (+170506us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 16:44:01.686401 (+ 622us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36068: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":145}
W20260926 16:44:01.686908 15732 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36068: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 16:44:01.687260 15758 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:01.507464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46263 (local address 127.0.0.1:36068)
0926 16:44:01.508483 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:01.508513 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:01.508733 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:01.509460 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:01.509466 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:01.509482 (+ 16us) client_negotiation.cc:189] Negotiated authn=JWT
0926 16:44:01.509846 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:01.509860 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:01.511236 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:01.511243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:01.513321 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:01.513340 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:01.514538 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:01.514546 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:01.514764 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:01.514831 (+ 67us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 16:44:01.686226 (+171395us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 16:44:01.686765 (+ 539us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.9.238.254:46263: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":809,"thread_start_us":519,"threads_started":1}
I20260926 16:44:01.689828 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 15663
I20260926 16:44:01.691869 15724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:01.820518 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 15663
2026-09-26T16:44:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (965 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (439 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (433 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:44:02 dist-test-slave-wt2j krb5kdc[15773](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:44:02 dist-test-slave-wt2j krb5kdc[15773](info): set up 2 sockets
Sep 26 16:44:02 dist-test-slave-wt2j krb5kdc[15773](info): commencing operation
krb5kdc: starting...
W20260926 16:44:03.873744 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:44:04 dist-test-slave-wt2j krb5kdc[15773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:44:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:44:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:44:04.140244 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:40853
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:34385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:40853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:04.487510 15786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:04.487933 15786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:04.488157 15786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:04.495959 15786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:44:04.496096 15786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:04.496143 15786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:04.496229 15786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:44:04.496305 15786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:44:04.506876 15786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:34385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:40853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:40853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:04.508399 15786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:04.509766 15786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:04.523351 15791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:04.523245 15794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:04.523295 15792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:04.525048 15786 server_base.cc:1034] running on GCE node
I20260926 16:44:04.525993 15786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:04.527729 15786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:04.529050 15786 hybrid_clock.cc:648] HybridClock initialized: now 1790441044528917 us; error 75 us; skew 500 ppm
I20260926 16:44:04.529461 15786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:04 dist-test-slave-wt2j krb5kdc[15773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441044, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:04.533119 15786 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:44:04.535048 15786 webserver.cc:460] Webserver started at http://127.9.238.254:33787/ using document root <none> and password file <none>
I20260926 16:44:04.535862 15786 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:04.536023 15786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:04.536350 15786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:04.540181 15786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "9b351dcd0b0044cdb302e1d6697a4c35"
format_stamp: "Formatted at 2026-09-26 16:44:04 on dist-test-slave-wt2j"
server_key: "94847e18d843bf19660d745f673f0916"
server_key_iv: "b3183c804178683f1078175dec3984eb"
server_key_version: "encryptionkey@0"
I20260926 16:44:04.541051 15786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "9b351dcd0b0044cdb302e1d6697a4c35"
format_stamp: "Formatted at 2026-09-26 16:44:04 on dist-test-slave-wt2j"
server_key: "94847e18d843bf19660d745f673f0916"
server_key_iv: "b3183c804178683f1078175dec3984eb"
server_key_version: "encryptionkey@0"
I20260926 16:44:04.547051 15786 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 16:44:04.551402 15801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:04.553382 15786 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:44:04.553653 15786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "9b351dcd0b0044cdb302e1d6697a4c35"
format_stamp: "Formatted at 2026-09-26 16:44:04 on dist-test-slave-wt2j"
server_key: "94847e18d843bf19660d745f673f0916"
server_key_iv: "b3183c804178683f1078175dec3984eb"
server_key_version: "encryptionkey@0"
I20260926 16:44:04.553894 15786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:04.574851 15786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:04.580221 15786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:04.580533 15786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:04.602921 15786 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:40853
I20260926 16:44:04.603025 15852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:40853 every 8 connection(s)
I20260926 16:44:04.604533 15786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:44:04.606559 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 15786
I20260926 16:44:04.606909 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:44:04.607880 10171 external_mini_cluster.cc:1428] Setting key beae5432f26995334c275e754d15233c
I20260926 16:44:04.611887 15853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 16:44:04 dist-test-slave-wt2j krb5kdc[15773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:44:04.631050 15853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b351dcd0b0044cdb302e1d6697a4c35: Bootstrap starting.
I20260926 16:44:04.636428 15853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b351dcd0b0044cdb302e1d6697a4c35: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:04.638235 15853 log.cc:824] T 00000000000000000000000000000000 P 9b351dcd0b0044cdb302e1d6697a4c35: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:04.640337 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:04.610680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48962 (local address 127.9.238.254:40853)
0926 16:44:04.611817 (+ 1137us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:04.611832 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:04.611890 (+ 58us) server_negotiation.cc:407] Connection header received
0926 16:44:04.614534 (+ 2644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:04.614567 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:04.615107 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:04.615624 (+ 517us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:04.618013 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:04.619706 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:04.622213 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:04.623092 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:04.623280 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:04.629458 (+ 6178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:04.629474 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:04.629504 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:04.629572 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:04.634285 (+ 4713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:04.635201 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:04.635213 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:04.635224 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:04.635481 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:04.635940 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:04.635945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:04.635948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:04.636938 (+ 990us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:04.637310 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:04.637768 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 16:44:04.638393 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":888,"thread_start_us":468,"threads_started":1}
I20260926 16:44:04.642851 15853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b351dcd0b0044cdb302e1d6697a4c35: No bootstrap required, opened a new log
I20260926 16:44:04.652911 15853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b351dcd0b0044cdb302e1d6697a4c35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b351dcd0b0044cdb302e1d6697a4c35" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40853 } }
I20260926 16:44:04.653719 15853 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b351dcd0b0044cdb302e1d6697a4c35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:04.653888 15853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b351dcd0b0044cdb302e1d6697a4c35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b351dcd0b0044cdb302e1d6697a4c35, State: Initialized, Role: FOLLOWER
I20260926 16:44:04.654767 15853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b351dcd0b0044cdb302e1d6697a4c35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b351dcd0b0044cdb302e1d6697a4c35" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40853 } }
I20260926 16:44:04.655041 15853 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b351dcd0b0044cdb302e1d6697a4c35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:44:04.655194 15853 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b351dcd0b0044cdb302e1d6697a4c35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:44:04.655434 15853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b351dcd0b0044cdb302e1d6697a4c35 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:04.658449 15853 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b351dcd0b0044cdb302e1d6697a4c35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b351dcd0b0044cdb302e1d6697a4c35" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40853 } }
I20260926 16:44:04.659292 15853 leader_election.cc:302] T 00000000000000000000000000000000 P 9b351dcd0b0044cdb302e1d6697a4c35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b351dcd0b0044cdb302e1d6697a4c35; no voters:
I20260926 16:44:04.660092 15853 leader_election.cc:288] T 00000000000000000000000000000000 P 9b351dcd0b0044cdb302e1d6697a4c35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:44:04.660317 15858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b351dcd0b0044cdb302e1d6697a4c35 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:44:04.661798 15858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b351dcd0b0044cdb302e1d6697a4c35 [term 1 LEADER]: Becoming Leader. State: Replica: 9b351dcd0b0044cdb302e1d6697a4c35, State: Running, Role: LEADER
I20260926 16:44:04.662493 15858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b351dcd0b0044cdb302e1d6697a4c35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b351dcd0b0044cdb302e1d6697a4c35" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40853 } }
I20260926 16:44:04.663163 15853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b351dcd0b0044cdb302e1d6697a4c35 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:44:04.667459 15859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b351dcd0b0044cdb302e1d6697a4c35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b351dcd0b0044cdb302e1d6697a4c35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b351dcd0b0044cdb302e1d6697a4c35" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40853 } } }
I20260926 16:44:04.667450 15860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b351dcd0b0044cdb302e1d6697a4c35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b351dcd0b0044cdb302e1d6697a4c35. Latest consensus state: current_term: 1 leader_uuid: "9b351dcd0b0044cdb302e1d6697a4c35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b351dcd0b0044cdb302e1d6697a4c35" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40853 } } }
I20260926 16:44:04.668047 15859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b351dcd0b0044cdb302e1d6697a4c35 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:04.668141 15860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b351dcd0b0044cdb302e1d6697a4c35 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:04.670542 15866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:44:04.677227 15866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:44:04.687922 15866 catalog_manager.cc:1347] Generated new cluster ID: d24a4f92b6e94e6290baacdb9878f9be
I20260926 16:44:04.688071 15866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:44:04.699832 15866 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:44:04.701388 15866 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:44:04.721225 15866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b351dcd0b0044cdb302e1d6697a4c35: Generated new TSK 0
I20260926 16:44:04.722740 15866 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
I20260926 16:44:04.788368 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:0
--local_ip_for_outbound_sockets=127.9.238.193
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:40853
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:34385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:05.138273 15879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:05.138692 15879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:05.138870 15879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:05.148475 15879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:05.148649 15879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:05.148856 15879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:44:05.162451 15879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:34385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:40853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:05.164158 15879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:05.166199 15879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:05.182327 15887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:05.182425 15884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:05.182425 15885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:05.183477 15879 server_base.cc:1034] running on GCE node
I20260926 16:44:05.184365 15879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:05.185664 15879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:05.186980 15879 hybrid_clock.cc:648] HybridClock initialized: now 1790441045186857 us; error 104 us; skew 500 ppm
I20260926 16:44:05.187484 15879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:05 dist-test-slave-wt2j krb5kdc[15773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441045, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:05.191411 15879 init.cc:377] Logged in from keytab as kudu/127.9.238.193@KRBTEST.COM (short username kudu)
I20260926 16:44:05.193289 15879 webserver.cc:460] Webserver started at http://127.9.238.193:38173/ using document root <none> and password file <none>
I20260926 16:44:05.194195 15879 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:05.194375 15879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:05.194743 15879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:05.198202 15879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-0/data/instance:
uuid: "42ddfcaa27f44048b18392ba84f560c9"
format_stamp: "Formatted at 2026-09-26 16:44:05 on dist-test-slave-wt2j"
server_key: "90e61c2376228bf42d469a782580536d"
server_key_iv: "7906a1635e8a2ffe439ce1f19489ec3b"
server_key_version: "encryptionkey@0"
I20260926 16:44:05.199134 15879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance:
uuid: "42ddfcaa27f44048b18392ba84f560c9"
format_stamp: "Formatted at 2026-09-26 16:44:05 on dist-test-slave-wt2j"
server_key: "90e61c2376228bf42d469a782580536d"
server_key_iv: "7906a1635e8a2ffe439ce1f19489ec3b"
server_key_version: "encryptionkey@0"
I20260926 16:44:05.205397 15879 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 16:44:05.209753 15894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:05.211781 15879 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 16:44:05.211989 15879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "42ddfcaa27f44048b18392ba84f560c9"
format_stamp: "Formatted at 2026-09-26 16:44:05 on dist-test-slave-wt2j"
server_key: "90e61c2376228bf42d469a782580536d"
server_key_iv: "7906a1635e8a2ffe439ce1f19489ec3b"
server_key_version: "encryptionkey@0"
I20260926 16:44:05.212225 15879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:05.235114 15879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:05.240032 15879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:05.240350 15879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:05.242508 15879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:05.244023 15879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:05.244146 15879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:05.244284 15879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:05.244357 15879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:05.284298 15879 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:38723
I20260926 16:44:05.284396 16006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:38723 every 8 connection(s)
I20260926 16:44:05.285640 15879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:44:05.289922 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 15879
I20260926 16:44:05.290196 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance
I20260926 16:44:05.291164 10171 external_mini_cluster.cc:1428] Setting key bacc36095c08a1de076cb0520faa7947
Sep 26 16:44:05 dist-test-slave-wt2j krb5kdc[15773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441045, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:44:05.311199 16009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:05.289182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:60683 (local address 127.9.238.254:40853)
0926 16:44:05.289745 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:05.289750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:05.290977 (+ 1227us) server_negotiation.cc:407] Connection header received
0926 16:44:05.293437 (+ 2460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:05.293445 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:05.293564 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:05.293677 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:05.295606 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:05.296941 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:05.298726 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:05.299521 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:05.299715 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:05.304161 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:05.304167 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:05.304171 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:05.304214 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:05.307298 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:05.308483 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:05.308487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:05.308490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:05.308622 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:05.309210 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:05.309213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:05.309216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:05.309535 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:05.309659 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:05.310675 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0926 16:44:05.310886 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":334,"threads_started":1}
I20260926 16:44:05.312879 16010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:05.289702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40853 (local address 127.9.238.193:60683)
0926 16:44:05.290776 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:05.290831 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:05.292926 (+ 2095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:05.293991 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:05.294003 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:05.294738 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:05.295362 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:05.295385 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:05.297187 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:05.297192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:05.298552 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:05.298564 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:05.299810 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:05.299815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:05.300040 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:05.300877 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:05.300902 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:05.303859 (+ 2957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:05.307506 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:05.307520 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:05.307543 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:05.308303 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:05.308753 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:05.308761 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:05.308767 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:05.309054 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:05.309685 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:05.309697 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:05.310229 (+ 532us) client_negotiation.cc:769] Sending connection context
0926 16:44:05.310561 (+ 332us) client_negotiation.cc:240] Negotiation successful
0926 16:44:05.311221 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":470,"threads_started":1}
I20260926 16:44:05.314906 16007 heartbeater.cc:347] Connected to a master server at 127.9.238.254:40853
I20260926 16:44:05.315310 16007 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.9.238.194@KRBTEST.COM; defaulting to no policy
I20260926 16:44:05.316429 16007 heartbeater.cc:511] Master 127.9.238.254:40853 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.194@KRBTEST.COM" created.
I20260926 16:44:05.319267 15818 ts_manager.cc:194] Registered new tserver with Master: 42ddfcaa27f44048b18392ba84f560c9 (127.9.238.193:38723)
I20260926 16:44:05.321806 15818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.193@KRBTEST.COM'} at 127.9.238.193:60683
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
I20260926 16:44:05.351540 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:0
--local_ip_for_outbound_sockets=127.9.238.194
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:40853
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:34385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:05.683254 16013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:05.683630 16013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:05.683806 16013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:05.691640 16013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:05.691777 16013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:05.691980 16013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:44:05.702809 16013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:34385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:40853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:05.704262 16013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:05.705668 16013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:05.721218 16021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:05.721269 16018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:05.721292 16019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:05.722661 16013 server_base.cc:1034] running on GCE node
I20260926 16:44:05.723604 16013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:05.725067 16013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:05.726428 16013 hybrid_clock.cc:648] HybridClock initialized: now 1790441045726333 us; error 74 us; skew 500 ppm
I20260926 16:44:05.726881 16013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:05 dist-test-slave-wt2j krb5kdc[15773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441045, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:05.731060 16013 init.cc:377] Logged in from keytab as kudu/127.9.238.194@KRBTEST.COM (short username kudu)
I20260926 16:44:05.733223 16013 webserver.cc:460] Webserver started at http://127.9.238.194:40623/ using document root <none> and password file <none>
I20260926 16:44:05.734175 16013 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:05.734331 16013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:05.734737 16013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:05.738428 16013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-1/data/instance:
uuid: "842f3a0b0aef4cad9f0c0c9b2b4a508c"
format_stamp: "Formatted at 2026-09-26 16:44:05 on dist-test-slave-wt2j"
server_key: "d1be4406fb6a0d383eaa3787583255fa"
server_key_iv: "50d4c57877d81947a3437a26ede8baac"
server_key_version: "encryptionkey@0"
I20260926 16:44:05.739363 16013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance:
uuid: "842f3a0b0aef4cad9f0c0c9b2b4a508c"
format_stamp: "Formatted at 2026-09-26 16:44:05 on dist-test-slave-wt2j"
server_key: "d1be4406fb6a0d383eaa3787583255fa"
server_key_iv: "50d4c57877d81947a3437a26ede8baac"
server_key_version: "encryptionkey@0"
I20260926 16:44:05.746040 16013 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 16:44:05.750838 16028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:05.753003 16013 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 16:44:05.753330 16013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "842f3a0b0aef4cad9f0c0c9b2b4a508c"
format_stamp: "Formatted at 2026-09-26 16:44:05 on dist-test-slave-wt2j"
server_key: "d1be4406fb6a0d383eaa3787583255fa"
server_key_iv: "50d4c57877d81947a3437a26ede8baac"
server_key_version: "encryptionkey@0"
I20260926 16:44:05.753602 16013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:05.776052 16013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:05.782040 16013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:05.782476 16013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:05.783911 16013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:05.785741 16013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:05.785831 16013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:05.785987 16013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:05.786041 16013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:05.828105 16013 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:35323
I20260926 16:44:05.828217 16140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:35323 every 8 connection(s)
I20260926 16:44:05.829916 16013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:44:05.837402 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 16013
I20260926 16:44:05.837617 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance
I20260926 16:44:05.838303 10171 external_mini_cluster.cc:1428] Setting key fb946e2cd140271214801dad72187fd0
Sep 26 16:44:05 dist-test-slave-wt2j krb5kdc[15773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441045, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:44:05.857247 16143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:05.833187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:55013 (local address 127.9.238.254:40853)
0926 16:44:05.833823 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:05.833828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:05.835104 (+ 1276us) server_negotiation.cc:407] Connection header received
0926 16:44:05.837596 (+ 2492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:05.837606 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:05.837737 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:05.837946 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:05.840485 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:05.841409 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:05.844453 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:05.844947 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:05.845071 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:05.850496 (+ 5425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:05.850503 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:05.850507 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:05.850546 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:05.853962 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:05.854742 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:05.854748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:05.854752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:05.854884 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:05.855334 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:05.855339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:05.855342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:05.855695 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:05.855860 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:05.856610 (+ 750us) server_negotiation.cc:299] Negotiation successful
0926 16:44:05.856848 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":288,"threads_started":1}
I20260926 16:44:05.858302 16144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:05.833910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40853 (local address 127.9.238.194:55013)
0926 16:44:05.834879 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:05.834932 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:05.837098 (+ 2166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:05.838324 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:05.838340 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:05.839258 (+ 918us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:05.840230 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:05.840261 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:05.841612 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:05.841618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:05.844249 (+ 2631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:05.844272 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:05.845254 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:05.845265 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:05.845653 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:05.846668 (+ 1015us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:05.846707 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:05.850173 (+ 3466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:05.854127 (+ 3954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:05.854137 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:05.854155 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:05.854605 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:05.855001 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:05.855008 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:05.855013 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:05.855205 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:05.855820 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:05.855829 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:05.856214 (+ 385us) client_negotiation.cc:769] Sending connection context
0926 16:44:05.856517 (+ 303us) client_negotiation.cc:240] Negotiation successful
0926 16:44:05.857073 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":372,"threads_started":1}
I20260926 16:44:05.860203 16141 heartbeater.cc:347] Connected to a master server at 127.9.238.254:40853
I20260926 16:44:05.860532 16141 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:05.861307 16141 heartbeater.cc:511] Master 127.9.238.254:40853 requested a full tablet report, sending...
I20260926 16:44:05.863051 15818 ts_manager.cc:194] Registered new tserver with Master: 842f3a0b0aef4cad9f0c0c9b2b4a508c (127.9.238.194:35323)
I20260926 16:44:05.864046 15818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.194@KRBTEST.COM'} at 127.9.238.194:55013
WARNING: no policy specified for kudu/127.9.238.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
I20260926 16:44:05.900061 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:0
--local_ip_for_outbound_sockets=127.9.238.195
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:40853
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:34385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:06.290454 16147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:06.290759 16147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:06.290861 16147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:06.298581 16147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:06.298704 16147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:06.298820 16147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:44:06.309378 16147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:34385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:40853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:06.310747 16147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:06.312132 16147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 16:44:06.325806 16007 heartbeater.cc:503] Master 127.9.238.254:40853 was elected leader, sending a full tablet report...
W20260926 16:44:06.328321 16155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:06.328446 16152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:06.329082 16153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:06.329731 16147 server_base.cc:1034] running on GCE node
I20260926 16:44:06.330556 16147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:06.331846 16147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:06.333132 16147 hybrid_clock.cc:648] HybridClock initialized: now 1790441046333072 us; error 48 us; skew 500 ppm
I20260926 16:44:06.333717 16147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:06 dist-test-slave-wt2j krb5kdc[15773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441046, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:06.338037 16147 init.cc:377] Logged in from keytab as kudu/127.9.238.195@KRBTEST.COM (short username kudu)
I20260926 16:44:06.340097 16147 webserver.cc:460] Webserver started at http://127.9.238.195:33141/ using document root <none> and password file <none>
I20260926 16:44:06.340924 16147 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:06.341087 16147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:06.341423 16147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:06.344730 16147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-2/data/instance:
uuid: "e36059588a714d5694cf54e7fdbd2917"
format_stamp: "Formatted at 2026-09-26 16:44:06 on dist-test-slave-wt2j"
server_key: "c92625d4d48a4b8e899cabcaf01f2422"
server_key_iv: "198f363272d52917844d468adf29e208"
server_key_version: "encryptionkey@0"
I20260926 16:44:06.345536 16147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance:
uuid: "e36059588a714d5694cf54e7fdbd2917"
format_stamp: "Formatted at 2026-09-26 16:44:06 on dist-test-slave-wt2j"
server_key: "c92625d4d48a4b8e899cabcaf01f2422"
server_key_iv: "198f363272d52917844d468adf29e208"
server_key_version: "encryptionkey@0"
I20260926 16:44:06.351635 16147 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 16:44:06.356053 16162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:06.358181 16147 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260926 16:44:06.358443 16147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "e36059588a714d5694cf54e7fdbd2917"
format_stamp: "Formatted at 2026-09-26 16:44:06 on dist-test-slave-wt2j"
server_key: "c92625d4d48a4b8e899cabcaf01f2422"
server_key_iv: "198f363272d52917844d468adf29e208"
server_key_version: "encryptionkey@0"
I20260926 16:44:06.358711 16147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:06.382004 16147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:06.387197 16147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:06.387462 16147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:06.388589 16147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:06.390000 16147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:06.390091 16147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:06.390179 16147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:06.390228 16147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:06.429247 16147 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:44055
I20260926 16:44:06.429427 16274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:44055 every 8 connection(s)
I20260926 16:44:06.430732 16147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:44:06.436458 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 16147
I20260926 16:44:06.436666 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance
I20260926 16:44:06.437242 10171 external_mini_cluster.cc:1428] Setting key e30c0ffefea061a4a3b681e0da350e08
Sep 26 16:44:06 dist-test-slave-wt2j krb5kdc[15773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441046, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:44:06.456553 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:06.434076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:41579 (local address 127.9.238.254:40853)
0926 16:44:06.434604 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:06.434649 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:06.436050 (+ 1401us) server_negotiation.cc:407] Connection header received
0926 16:44:06.439512 (+ 3462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:06.439516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:06.439681 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:06.439829 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:06.442051 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:06.442889 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:06.444947 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:06.445441 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:06.445577 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:06.449763 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:06.449768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:06.449772 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:06.449810 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:06.453013 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:06.454048 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:06.454054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:06.454058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:06.454188 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:06.454654 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:06.454660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:06.454668 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:06.454925 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:06.455081 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:06.455938 (+ 857us) server_negotiation.cc:299] Negotiation successful
0926 16:44:06.456162 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":286,"threads_started":1}
I20260926 16:44:06.457682 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:06.434554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40853 (local address 127.9.238.195:41579)
0926 16:44:06.435757 (+ 1203us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:06.435828 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:06.439092 (+ 3264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:06.440188 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:06.440199 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:06.440957 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:06.441826 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:06.441853 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:06.443136 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:06.443141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:06.444758 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:06.444773 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:06.445699 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:06.445704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:06.445970 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:06.446780 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:06.446811 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:06.449493 (+ 2682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:06.453208 (+ 3715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:06.453221 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:06.453250 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:06.453904 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:06.454304 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:06.454309 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:06.454313 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:06.454528 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:06.455057 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:06.455065 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:06.455466 (+ 401us) client_negotiation.cc:769] Sending connection context
0926 16:44:06.455834 (+ 368us) client_negotiation.cc:240] Negotiation successful
0926 16:44:06.456444 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":434,"threads_started":1}
I20260926 16:44:06.459595 16275 heartbeater.cc:347] Connected to a master server at 127.9.238.254:40853
I20260926 16:44:06.460059 16275 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:06.461010 16275 heartbeater.cc:511] Master 127.9.238.254:40853 requested a full tablet report, sending...
I20260926 16:44:06.462719 15818 ts_manager.cc:194] Registered new tserver with Master: e36059588a714d5694cf54e7fdbd2917 (127.9.238.195:44055)
I20260926 16:44:06.463766 15818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.195@KRBTEST.COM'} at 127.9.238.195:41579
I20260926 16:44:06.467764 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/test-admin.keytab.
Sep 26 16:44:06 dist-test-slave-wt2j krb5kdc[15773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:06.503173 10171 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 16:44:06.510120 16283 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:06.506942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36234 (local address 127.9.238.193:38723)
0926 16:44:06.507923 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:06.507931 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:06.507982 (+ 51us) server_negotiation.cc:407] Connection header received
0926 16:44:06.508063 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:06.508070 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:06.508310 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:06.508498 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:06.509697 (+ 1199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36234: BlockingRecv error: recv got EOF from 127.0.0.1:36234 (error 108)
Metrics: {"server-negotiator.queue_time_us":802,"thread_start_us":582,"threads_started":1}
W20260926 16:44:06.510404 10171 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/krb5cc' of type FILE
Sep 26 16:44:06 dist-test-slave-wt2j krb5kdc[15773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:06.512543 10171 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 16:44:06 dist-test-slave-wt2j krb5kdc[15773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:06.513995 10171 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 16:44:06 dist-test-slave-wt2j krb5kdc[15773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.193@KRBTEST.COM
I20260926 16:44:06.532042 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:06.516078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36240 (local address 127.9.238.193:38723)
0926 16:44:06.516339 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:06.516344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:06.516774 (+ 430us) server_negotiation.cc:407] Connection header received
0926 16:44:06.517014 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:06.517018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:06.517086 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:06.517197 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:06.518413 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:06.519489 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:06.520940 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:06.521562 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:06.521717 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:06.526078 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:06.526102 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:06.526119 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:06.526166 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:06.529238 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:06.529945 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:06.529959 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:06.529961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:06.530083 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:06.530438 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:06.530459 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:06.530462 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:06.530819 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:06.531114 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:06.531461 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 16:44:06.531743 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
W20260926 16:44:06.539391 16283 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:06.537262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36244 (local address 127.9.238.193:38723)
0926 16:44:06.537566 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:06.537571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:06.537668 (+ 97us) server_negotiation.cc:407] Connection header received
0926 16:44:06.537819 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:06.537824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:06.537986 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:06.538103 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:06.539031 (+ 928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36244: BlockingRecv error: recv got EOF from 127.0.0.1:36244 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260926 16:44:06.539455 10171 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/krb5cc' of type FILE
Sep 26 16:44:06 dist-test-slave-wt2j krb5kdc[15773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:06.541473 10171 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 16:44:06 dist-test-slave-wt2j krb5kdc[15773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:06.543087 10171 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 16:44:06 dist-test-slave-wt2j krb5kdc[15773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.193@KRBTEST.COM
I20260926 16:44:06.560904 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:06.544919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36254 (local address 127.9.238.193:38723)
0926 16:44:06.545217 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:06.545226 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:06.545415 (+ 189us) server_negotiation.cc:407] Connection header received
0926 16:44:06.545583 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:06.545587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:06.545657 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:06.545756 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:06.546847 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:06.547718 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:06.549486 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:06.549941 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:06.550108 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:06.554722 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:06.554727 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:06.554731 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:06.554784 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:06.557714 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:06.558707 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:06.558711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:06.558713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:06.558855 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:06.559441 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:06.559444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:06.559446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:06.559843 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:06.560049 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:06.560405 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 16:44:06.560583 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
W20260926 16:44:06.566907 16283 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:06.564742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36260 (local address 127.9.238.193:38723)
0926 16:44:06.564983 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:06.564988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:06.565338 (+ 350us) server_negotiation.cc:407] Connection header received
0926 16:44:06.565509 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:06.565513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:06.565593 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:06.565708 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:06.566645 (+ 937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36260: BlockingRecv error: recv got EOF from 127.0.0.1:36260 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260926 16:44:06.567183 10171 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790440988577176-10171-0/krb5kdc/krb5cc' of type FILE
Sep 26 16:44:06 dist-test-slave-wt2j krb5kdc[15773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:06.569362 10171 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 16:44:06 dist-test-slave-wt2j krb5kdc[15773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:06.571099 10171 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 16:44:06 dist-test-slave-wt2j krb5kdc[15773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.193@KRBTEST.COM
I20260926 16:44:06.590730 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:06.573233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36270 (local address 127.9.238.193:38723)
0926 16:44:06.573534 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:06.573543 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:06.573795 (+ 252us) server_negotiation.cc:407] Connection header received
0926 16:44:06.574058 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:06.574064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:06.574146 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:06.574262 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:06.575426 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:06.576436 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:06.577793 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:06.578269 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:06.578425 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:06.584554 (+ 6129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:06.584561 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:06.584565 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:06.584615 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:06.587468 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:06.588674 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:06.588678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:06.588681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:06.588855 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:06.589238 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:06.589242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:06.589244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:06.589561 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:06.589804 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:06.590094 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 16:44:06.590294 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260926 16:44:06.592728 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 15879
I20260926 16:44:06.607620 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:06.593643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36272 (local address 127.9.238.193:38723)
0926 16:44:06.593847 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:06.593852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:06.594331 (+ 479us) server_negotiation.cc:407] Connection header received
0926 16:44:06.594490 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:06.594494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:06.594570 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:06.594663 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:06.595800 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:06.596550 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:06.598004 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:06.598459 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:06.598572 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:06.601188 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:06.601194 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:06.601198 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:06.601248 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:06.604338 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:06.605213 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:06.605217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:06.605220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:06.605417 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:06.605849 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:06.605852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:06.605854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:06.606181 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:06.606416 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:06.606860 (+ 444us) server_negotiation.cc:299] Negotiation successful
0926 16:44:06.607098 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 16:44:06.608345 16002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:06.701936 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 15879
I20260926 16:44:06.722546 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 16013
Sep 26 16:44:06 dist-test-slave-wt2j krb5kdc[15773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.194@KRBTEST.COM
I20260926 16:44:06.740974 16296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:06.723693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38788 (local address 127.9.238.194:35323)
0926 16:44:06.724381 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:06.724390 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:06.724420 (+ 30us) server_negotiation.cc:407] Connection header received
0926 16:44:06.724503 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:06.724511 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:06.724775 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:06.724993 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:06.726062 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:06.727181 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:06.728883 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:06.729558 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:06.729713 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:06.734231 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:06.734246 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:06.734267 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:06.734313 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:06.737750 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:06.738530 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:06.738538 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:06.738540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:06.738681 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:06.739072 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:06.739076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:06.739078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:06.739497 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:06.739907 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:06.740305 (+ 398us) server_negotiation.cc:299] Negotiation successful
0926 16:44:06.740606 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":330,"threads_started":1}
I20260926 16:44:06.742511 16136 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:06.830247 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 16013
I20260926 16:44:06.845597 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 16147
Sep 26 16:44:06 dist-test-slave-wt2j krb5kdc[15773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.195@KRBTEST.COM
I20260926 16:44:06.864143 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:06.846913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60312 (local address 127.9.238.195:44055)
0926 16:44:06.847863 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:06.847872 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:06.847905 (+ 33us) server_negotiation.cc:407] Connection header received
0926 16:44:06.847970 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:06.847977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:06.848262 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:06.848444 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:06.849578 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:06.850568 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:06.851949 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:06.852523 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:06.852670 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:06.857730 (+ 5060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:06.857745 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:06.857761 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:06.857804 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:06.861036 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:06.861776 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:06.861784 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:06.861786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:06.861908 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:06.862354 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:06.862360 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:06.862363 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:06.862745 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:06.863097 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:06.863453 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 16:44:06.863775 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":769,"thread_start_us":493,"threads_started":1}
I20260926 16:44:06.866142 16270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:06.959532 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 16147
I20260926 16:44:06.976490 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 15786
I20260926 16:44:06.978066 15848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:07.083961 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 15786
2026-09-26T16:44:07Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4399 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:44:07 dist-test-slave-wt2j krb5kdc[16303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:44:07 dist-test-slave-wt2j krb5kdc[16303](info): set up 2 sockets
Sep 26 16:44:07 dist-test-slave-wt2j krb5kdc[16303](info): commencing operation
krb5kdc: starting...
W20260926 16:44:08.269949 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:44:08 dist-test-slave-wt2j krb5kdc[16303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:44:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:44:08Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.254.keytab.
Entry for principal oryx/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.254.keytab.
I20260926 16:44:08.529021 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:40511
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.254.keytab
--principal=oryx/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:39619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:40511
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:08.947127 16316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:08.947516 16316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:08.947731 16316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:08.956463 16316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:44:08.956626 16316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:08.956741 16316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:08.956884 16316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:44:08.956950 16316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:44:08.968310 16316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:39619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:40511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.254.keytab
--principal=oryx/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:40511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:08.970165 16316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:08.971904 16316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:08.987442 16324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:08.987406 16322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:08.987902 16321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:08.988633 16316 server_base.cc:1034] running on GCE node
I20260926 16:44:08.989665 16316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:08.991761 16316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:08.993036 16316 hybrid_clock.cc:648] HybridClock initialized: now 1790441048992988 us; error 39 us; skew 500 ppm
I20260926 16:44:08.993492 16316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:08 dist-test-slave-wt2j krb5kdc[16303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441048, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:09.004388 16316 init.cc:377] Logged in from keytab as oryx/127.9.238.254@KRBTEST.COM (short username oryx)
I20260926 16:44:09.006423 16316 webserver.cc:460] Webserver started at http://127.9.238.254:35795/ using document root <none> and password file <none>
I20260926 16:44:09.007191 16316 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:09.007304 16316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:09.007650 16316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:09.010833 16316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "212c6cc5c6e6456b919eb5680dc91f11"
format_stamp: "Formatted at 2026-09-26 16:44:09 on dist-test-slave-wt2j"
server_key: "d1abf948d81d89bc4c364541ef5ee282"
server_key_iv: "bb66be9eb4fa0d06b230df12f900f1aa"
server_key_version: "encryptionkey@0"
I20260926 16:44:09.011633 16316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "212c6cc5c6e6456b919eb5680dc91f11"
format_stamp: "Formatted at 2026-09-26 16:44:09 on dist-test-slave-wt2j"
server_key: "d1abf948d81d89bc4c364541ef5ee282"
server_key_iv: "bb66be9eb4fa0d06b230df12f900f1aa"
server_key_version: "encryptionkey@0"
I20260926 16:44:09.017596 16316 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260926 16:44:09.021953 16331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:09.023866 16316 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 16:44:09.024083 16316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "212c6cc5c6e6456b919eb5680dc91f11"
format_stamp: "Formatted at 2026-09-26 16:44:09 on dist-test-slave-wt2j"
server_key: "d1abf948d81d89bc4c364541ef5ee282"
server_key_iv: "bb66be9eb4fa0d06b230df12f900f1aa"
server_key_version: "encryptionkey@0"
I20260926 16:44:09.024289 16316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:09.049625 16316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:09.054243 16316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:09.054534 16316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:09.075786 16316 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:40511
I20260926 16:44:09.075881 16382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:40511 every 8 connection(s)
I20260926 16:44:09.077317 16316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:44:09.082402 16383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:09.086268 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 16316
I20260926 16:44:09.086546 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:44:09.087285 10171 external_mini_cluster.cc:1428] Setting key fb81d362f237a396661c6f6bc574c8a8
I20260926 16:44:09.099112 16383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11: Bootstrap starting.
I20260926 16:44:09.104189 16383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11: Neither blocks nor log segments found. Creating new log.
Sep 26 16:44:09 dist-test-slave-wt2j krb5kdc[16303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM
I20260926 16:44:09.105701 16383 log.cc:824] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:09.110256 16383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11: No bootstrap required, opened a new log
I20260926 16:44:09.115183 16386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:09.090147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48774 (local address 127.9.238.254:40511)
0926 16:44:09.091195 (+ 1048us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:09.091216 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:09.091283 (+ 67us) server_negotiation.cc:407] Connection header received
0926 16:44:09.093705 (+ 2422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:09.093733 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:09.094226 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:09.094659 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:09.096921 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:09.098353 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:09.100343 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:09.101303 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:09.101477 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:09.107007 (+ 5530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:09.107017 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:09.107041 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:09.107135 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:09.110725 (+ 3590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:09.111636 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:09.111652 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:09.111660 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:09.111832 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:09.112223 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:09.112227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:09.112229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:09.112802 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:09.113181 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:09.113582 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 16:44:09.114033 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":466,"threads_started":1}
I20260926 16:44:09.120898 16383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } }
I20260926 16:44:09.121490 16383 raft_consensus.cc:374] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:09.121631 16383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212c6cc5c6e6456b919eb5680dc91f11, State: Initialized, Role: FOLLOWER
I20260926 16:44:09.122359 16383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } }
I20260926 16:44:09.122591 16383 raft_consensus.cc:388] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:44:09.122772 16383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:44:09.122991 16383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:09.125761 16383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } }
I20260926 16:44:09.126428 16383 leader_election.cc:302] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 212c6cc5c6e6456b919eb5680dc91f11; no voters:
I20260926 16:44:09.127229 16383 leader_election.cc:288] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:44:09.127470 16388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:44:09.128633 16388 raft_consensus.cc:686] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [term 1 LEADER]: Becoming Leader. State: Replica: 212c6cc5c6e6456b919eb5680dc91f11, State: Running, Role: LEADER
I20260926 16:44:09.129566 16388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } }
I20260926 16:44:09.131690 16383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:44:09.133589 16389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 212c6cc5c6e6456b919eb5680dc91f11. Latest consensus state: current_term: 1 leader_uuid: "212c6cc5c6e6456b919eb5680dc91f11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } }
I20260926 16:44:09.133958 16390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "212c6cc5c6e6456b919eb5680dc91f11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } }
I20260926 16:44:09.136632 16389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:09.136749 16390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:09.141557 16395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:44:09.149114 16395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:44:09.160061 16395 catalog_manager.cc:1347] Generated new cluster ID: 58607067b23c4cbca0a242e80249d2d2
I20260926 16:44:09.160225 16395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:44:09.176329 16395 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:44:09.178426 16395 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:44:09.187103 16395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11: Generated new TSK 0
I20260926 16:44:09.188249 16395 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.193.keytab.
Entry for principal oryx/127.9.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.193.keytab.
I20260926 16:44:09.253260 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:0
--local_ip_for_outbound_sockets=127.9.238.193
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:40511
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.193.keytab
--principal=oryx/127.9.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:39619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:09.672366 16409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:09.672797 16409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:09.672998 16409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:09.681412 16409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:09.681618 16409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:09.681830 16409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:44:09.693424 16409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:39619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.193.keytab
--principal=oryx/127.9.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:40511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:09.695201 16409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:09.696949 16409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:09.712824 16414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:09.712870 16417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:09.712965 16415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:09.714048 16409 server_base.cc:1034] running on GCE node
I20260926 16:44:09.714986 16409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:09.716542 16409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:09.717859 16409 hybrid_clock.cc:648] HybridClock initialized: now 1790441049717798 us; error 47 us; skew 500 ppm
I20260926 16:44:09.718345 16409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:09 dist-test-slave-wt2j krb5kdc[16303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441049, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:09.722357 16409 init.cc:377] Logged in from keytab as oryx/127.9.238.193@KRBTEST.COM (short username oryx)
I20260926 16:44:09.724278 16409 webserver.cc:460] Webserver started at http://127.9.238.193:41751/ using document root <none> and password file <none>
I20260926 16:44:09.725143 16409 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:09.725301 16409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:09.725677 16409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:09.729519 16409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/data/instance:
uuid: "e7a44bfa35494003aadce77ec43974a8"
format_stamp: "Formatted at 2026-09-26 16:44:09 on dist-test-slave-wt2j"
server_key: "92b11cdd5b80c9d72d502792e4230658"
server_key_iv: "2f874528f41b89a587151517f37b5af5"
server_key_version: "encryptionkey@0"
I20260926 16:44:09.730443 16409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance:
uuid: "e7a44bfa35494003aadce77ec43974a8"
format_stamp: "Formatted at 2026-09-26 16:44:09 on dist-test-slave-wt2j"
server_key: "92b11cdd5b80c9d72d502792e4230658"
server_key_iv: "2f874528f41b89a587151517f37b5af5"
server_key_version: "encryptionkey@0"
I20260926 16:44:09.737308 16409 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 16:44:09.742347 16424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:09.744500 16409 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:44:09.744748 16409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "e7a44bfa35494003aadce77ec43974a8"
format_stamp: "Formatted at 2026-09-26 16:44:09 on dist-test-slave-wt2j"
server_key: "92b11cdd5b80c9d72d502792e4230658"
server_key_iv: "2f874528f41b89a587151517f37b5af5"
server_key_version: "encryptionkey@0"
I20260926 16:44:09.744998 16409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:09.768014 16409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:09.773530 16409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:09.773917 16409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:09.775279 16409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:09.777257 16409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:09.777356 16409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:09.777525 16409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:09.777602 16409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:09.818696 16409 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:40747
I20260926 16:44:09.818771 16536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:40747 every 8 connection(s)
I20260926 16:44:09.820427 16409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:44:09.821620 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 16409
I20260926 16:44:09.821825 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance
I20260926 16:44:09.822657 10171 external_mini_cluster.cc:1428] Setting key b89b36f771aae3fd077a0db8ce092c72
Sep 26 16:44:09 dist-test-slave-wt2j krb5kdc[16303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441049, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.9.238.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.238.194@KRBTEST.COM" created.
I20260926 16:44:09.849466 16539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:09.824295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:35135 (local address 127.9.238.254:40511)
0926 16:44:09.824816 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:09.824821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:09.826754 (+ 1933us) server_negotiation.cc:407] Connection header received
0926 16:44:09.830013 (+ 3259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:09.830020 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:09.830128 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:09.830440 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:09.833073 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:09.834322 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:09.836709 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:09.837168 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:09.837329 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:09.842098 (+ 4769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:09.842104 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:09.842108 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:09.842151 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:09.845102 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:09.846160 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:09.846171 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:09.846178 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:09.846384 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:09.846869 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:09.846873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:09.846875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:09.847215 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:09.847343 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:09.848656 (+ 1313us) server_negotiation.cc:299] Negotiation successful
0926 16:44:09.848955 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":304,"threads_started":1}
I20260926 16:44:09.850798 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:09.825263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.9.238.193:35135)
0926 16:44:09.826514 (+ 1251us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:09.826589 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:09.829523 (+ 2934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:09.830747 (+ 1224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:09.830764 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:09.831734 (+ 970us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:09.832733 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:09.832763 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:09.834616 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:09.834621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:09.836485 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:09.836505 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:09.837478 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:09.837486 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:09.837771 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:09.838649 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:09.838678 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:09.841749 (+ 3071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:09.845296 (+ 3547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:09.845308 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:09.845350 (+ 42us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:09.845884 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:09.846520 (+ 636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:09.846525 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:09.846527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:09.846744 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:09.847385 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:09.847393 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:09.848039 (+ 646us) client_negotiation.cc:769] Sending connection context
0926 16:44:09.848465 (+ 426us) client_negotiation.cc:240] Negotiation successful
0926 16:44:09.849173 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"thread_start_us":434,"threads_started":1}
I20260926 16:44:09.852898 16537 heartbeater.cc:347] Connected to a master server at 127.9.238.254:40511
I20260926 16:44:09.853286 16537 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:09.854239 16537 heartbeater.cc:511] Master 127.9.238.254:40511 requested a full tablet report, sending...
I20260926 16:44:09.857014 16348 ts_manager.cc:194] Registered new tserver with Master: e7a44bfa35494003aadce77ec43974a8 (127.9.238.193:40747)
I20260926 16:44:09.858803 16348 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.238.193@KRBTEST.COM'} at 127.9.238.193:35135
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.194.keytab.
Entry for principal oryx/127.9.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.194.keytab.
I20260926 16:44:09.876400 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:0
--local_ip_for_outbound_sockets=127.9.238.194
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:40511
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.194.keytab
--principal=oryx/127.9.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:39619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:10.300693 16543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:10.301020 16543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:10.301126 16543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:10.308437 16543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:10.308568 16543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:10.308732 16543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:44:10.318992 16543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:39619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.194.keytab
--principal=oryx/127.9.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:40511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:10.320480 16543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:10.321933 16543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:10.339028 16548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:10.338970 16551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:10.338999 16549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:10.340099 16543 server_base.cc:1034] running on GCE node
I20260926 16:44:10.340873 16543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:10.342020 16543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:10.343327 16543 hybrid_clock.cc:648] HybridClock initialized: now 1790441050343263 us; error 57 us; skew 500 ppm
I20260926 16:44:10.343813 16543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:10 dist-test-slave-wt2j krb5kdc[16303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441050, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:10.347962 16543 init.cc:377] Logged in from keytab as oryx/127.9.238.194@KRBTEST.COM (short username oryx)
I20260926 16:44:10.350073 16543 webserver.cc:460] Webserver started at http://127.9.238.194:33303/ using document root <none> and password file <none>
I20260926 16:44:10.350946 16543 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:10.351121 16543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:10.351548 16543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:10.355517 16543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/data/instance:
uuid: "b9822a16ad8646d78ff6a980fd5dc5b6"
format_stamp: "Formatted at 2026-09-26 16:44:10 on dist-test-slave-wt2j"
server_key: "77ba6c8eff7f6b7b11ba6a895e5aff93"
server_key_iv: "88bb0d771af4dfe15e40acbd4d8c5a57"
server_key_version: "encryptionkey@0"
I20260926 16:44:10.356577 16543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance:
uuid: "b9822a16ad8646d78ff6a980fd5dc5b6"
format_stamp: "Formatted at 2026-09-26 16:44:10 on dist-test-slave-wt2j"
server_key: "77ba6c8eff7f6b7b11ba6a895e5aff93"
server_key_iv: "88bb0d771af4dfe15e40acbd4d8c5a57"
server_key_version: "encryptionkey@0"
I20260926 16:44:10.363117 16543 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260926 16:44:10.367929 16558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:10.369870 16543 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260926 16:44:10.370175 16543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "b9822a16ad8646d78ff6a980fd5dc5b6"
format_stamp: "Formatted at 2026-09-26 16:44:10 on dist-test-slave-wt2j"
server_key: "77ba6c8eff7f6b7b11ba6a895e5aff93"
server_key_iv: "88bb0d771af4dfe15e40acbd4d8c5a57"
server_key_version: "encryptionkey@0"
I20260926 16:44:10.370397 16543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:10.390563 16543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:10.395196 16543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:10.395515 16543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:10.396718 16543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:10.398236 16543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:10.398342 16543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:10.398495 16543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:10.398569 16543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:10.437000 16543 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:34685
I20260926 16:44:10.437101 16670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:34685 every 8 connection(s)
I20260926 16:44:10.438385 16543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:44:10.445050 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 16543
I20260926 16:44:10.445228 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance
I20260926 16:44:10.445722 10171 external_mini_cluster.cc:1428] Setting key 5d9046a4d55541513b9040a37470d5b9
Sep 26 16:44:10 dist-test-slave-wt2j krb5kdc[16303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441050, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM
I20260926 16:44:10.469022 16673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:10.441695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:48133 (local address 127.9.238.254:40511)
0926 16:44:10.442289 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:10.442295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:10.444028 (+ 1733us) server_negotiation.cc:407] Connection header received
0926 16:44:10.446764 (+ 2736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:10.446771 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:10.446871 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:10.447089 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:10.449483 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:10.450837 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:10.452864 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:10.453620 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:10.453787 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:10.459533 (+ 5746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:10.459578 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:10.459587 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:10.459659 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:10.464368 (+ 4709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:10.465537 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:10.465548 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:10.465555 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:10.465780 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:10.466362 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:10.466371 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:10.466378 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:10.466917 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:10.467120 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:10.468073 (+ 953us) server_negotiation.cc:299] Negotiation successful
0926 16:44:10.468409 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":307,"threads_started":1}
I20260926 16:44:10.469682 16674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:10.442393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.9.238.194:48133)
0926 16:44:10.443782 (+ 1389us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:10.443842 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:10.446311 (+ 2469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:10.447420 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:10.447435 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:10.448483 (+ 1048us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:10.449271 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:10.449294 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:10.451056 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:10.451060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:10.452679 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:10.452693 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:10.454051 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:10.454058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:10.454425 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:10.455489 (+ 1064us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:10.455528 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:10.459045 (+ 3517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:10.464650 (+ 5605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:10.464660 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:10.464677 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:10.465319 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:10.465927 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:10.465930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:10.465933 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:10.466192 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:10.467106 (+ 914us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:10.467113 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:10.467501 (+ 388us) client_negotiation.cc:769] Sending connection context
0926 16:44:10.467873 (+ 372us) client_negotiation.cc:240] Negotiation successful
0926 16:44:10.468448 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":352,"threads_started":1}
I20260926 16:44:10.471652 16671 heartbeater.cc:347] Connected to a master server at 127.9.238.254:40511
I20260926 16:44:10.472208 16671 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:10.473664 16671 heartbeater.cc:511] Master 127.9.238.254:40511 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.9.238.195@KRBTEST.COM; defaulting to no policy
I20260926 16:44:10.476121 16348 ts_manager.cc:194] Registered new tserver with Master: b9822a16ad8646d78ff6a980fd5dc5b6 (127.9.238.194:34685)
I20260926 16:44:10.477257 16348 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.238.194@KRBTEST.COM'} at 127.9.238.194:48133
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.195.keytab.
Entry for principal oryx/127.9.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.195.keytab.
I20260926 16:44:10.509858 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:0
--local_ip_for_outbound_sockets=127.9.238.195
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:40511
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.195.keytab
--principal=oryx/127.9.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:39619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:44:10.862911 16537 heartbeater.cc:503] Master 127.9.238.254:40511 was elected leader, sending a full tablet report...
W20260926 16:44:10.865242 16677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:10.865581 16677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:10.865748 16677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:10.873183 16677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:10.873368 16677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:10.873607 16677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:44:10.884424 16677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:39619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.195.keytab
--principal=oryx/127.9.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:40511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:10.885805 16677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:10.887189 16677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:10.901881 16683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:10.901952 16682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:10.901973 16685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:10.902747 16677 server_base.cc:1034] running on GCE node
I20260926 16:44:10.903340 16677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:10.904474 16677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:10.905711 16677 hybrid_clock.cc:648] HybridClock initialized: now 1790441050905647 us; error 48 us; skew 500 ppm
I20260926 16:44:10.906123 16677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:10 dist-test-slave-wt2j krb5kdc[16303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441050, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:10.909698 16677 init.cc:377] Logged in from keytab as oryx/127.9.238.195@KRBTEST.COM (short username oryx)
I20260926 16:44:10.911294 16677 webserver.cc:460] Webserver started at http://127.9.238.195:33733/ using document root <none> and password file <none>
I20260926 16:44:10.912053 16677 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:10.912158 16677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:10.912396 16677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:10.915460 16677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/data/instance:
uuid: "ee01f03feda544c59ff2dd4f5467404b"
format_stamp: "Formatted at 2026-09-26 16:44:10 on dist-test-slave-wt2j"
server_key: "bb63c891024b81999442df12511dd200"
server_key_iv: "f45fa4866c9e88a17979020fdbc8696b"
server_key_version: "encryptionkey@0"
I20260926 16:44:10.916287 16677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance:
uuid: "ee01f03feda544c59ff2dd4f5467404b"
format_stamp: "Formatted at 2026-09-26 16:44:10 on dist-test-slave-wt2j"
server_key: "bb63c891024b81999442df12511dd200"
server_key_iv: "f45fa4866c9e88a17979020fdbc8696b"
server_key_version: "encryptionkey@0"
I20260926 16:44:10.921867 16677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:44:10.925751 16692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:10.927493 16677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 16:44:10.927927 16677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "ee01f03feda544c59ff2dd4f5467404b"
format_stamp: "Formatted at 2026-09-26 16:44:10 on dist-test-slave-wt2j"
server_key: "bb63c891024b81999442df12511dd200"
server_key_iv: "f45fa4866c9e88a17979020fdbc8696b"
server_key_version: "encryptionkey@0"
I20260926 16:44:10.928140 16677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:10.950299 16677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:10.954841 16677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:10.955143 16677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:10.956259 16677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:10.957700 16677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:10.957806 16677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:10.957949 16677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:10.958020 16677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:10.996428 16677 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:42613
I20260926 16:44:10.996451 16804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:42613 every 8 connection(s)
I20260926 16:44:10.997710 16677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:44:11.006709 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 16677
I20260926 16:44:11.007012 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance
I20260926 16:44:11.007625 10171 external_mini_cluster.cc:1428] Setting key 9149e2bb2861abb3be68f5387b37f82a
Sep 26 16:44:11 dist-test-slave-wt2j krb5kdc[16303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441050, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM
I20260926 16:44:11.019912 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:11.001132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:39777 (local address 127.9.238.254:40511)
0926 16:44:11.001714 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:11.001719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:11.002821 (+ 1102us) server_negotiation.cc:407] Connection header received
0926 16:44:11.004854 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:11.004859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:11.004970 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:11.005092 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:11.006642 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.007627 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.009562 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.010063 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.010199 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:11.013957 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:11.013962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:11.013965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:11.013998 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:11.016750 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:11.017498 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:11.017505 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:11.017508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:11.017644 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:11.018063 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:11.018071 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:11.018074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:11.018351 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:11.018512 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:11.019244 (+ 732us) server_negotiation.cc:299] Negotiation successful
0926 16:44:11.019464 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":293,"threads_started":1}
I20260926 16:44:11.020860 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:11.001813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.9.238.195:39777)
0926 16:44:11.002640 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:11.002685 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:11.004553 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:11.005346 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:11.005355 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:11.005883 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:11.006473 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:11.006490 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.007779 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.007785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:11.009398 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:11.009413 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.010291 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.010299 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:11.010553 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:11.011310 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:11.011333 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:11.013728 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:11.016908 (+ 3180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:11.016917 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:11.016935 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:11.017374 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:11.017754 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:11.017760 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:11.017764 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:11.017947 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:11.018481 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:11.018489 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:11.018862 (+ 373us) client_negotiation.cc:769] Sending connection context
0926 16:44:11.019158 (+ 296us) client_negotiation.cc:240] Negotiation successful
0926 16:44:11.019718 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":305,"threads_started":1}
I20260926 16:44:11.022933 16805 heartbeater.cc:347] Connected to a master server at 127.9.238.254:40511
I20260926 16:44:11.023494 16805 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:11.024770 16805 heartbeater.cc:511] Master 127.9.238.254:40511 requested a full tablet report, sending...
I20260926 16:44:11.026510 16348 ts_manager.cc:194] Registered new tserver with Master: ee01f03feda544c59ff2dd4f5467404b (127.9.238.195:42613)
I20260926 16:44:11.027374 16348 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.238.195@KRBTEST.COM'} at 127.9.238.195:39777
I20260926 16:44:11.037567 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.238.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.238.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.238.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.253.keytab.
Entry for principal oryx/127.9.238.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.253.keytab.
I20260926 16:44:11.096783 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.253:38171
--webserver_interface=127.9.238.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.253.keytab
--principal=oryx/127.9.238.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:39619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:40511,127.9.238.253:38171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:11.469026 16811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:11.469404 16811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:11.469583 16811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:11.477335 16811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:44:11.477485 16811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:11.477623 16811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:11.477720 16811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:44:11.477789 16811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:44:11.480849 16671 heartbeater.cc:503] Master 127.9.238.254:40511 was elected leader, sending a full tablet report...
I20260926 16:44:11.488425 16811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:39619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:40511,127.9.238.253:38171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.253.keytab
--principal=oryx/127.9.238.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.253:38171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.238.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
W20260926 16:44:11.488849 16811 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.9.238.254:40511,127.9.238.253:38171'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 16:44:11.489967 16811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:11.491323 16811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:11.505281 16819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:11.505465 16816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:11.505430 16817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:11.506460 16811 server_base.cc:1034] running on GCE node
I20260926 16:44:11.507294 16811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:11.508968 16811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:11.510321 16811 hybrid_clock.cc:648] HybridClock initialized: now 1790441051510208 us; error 80 us; skew 500 ppm
I20260926 16:44:11.510751 16811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:11 dist-test-slave-wt2j krb5kdc[16303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441051, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:11.514503 16811 init.cc:377] Logged in from keytab as oryx/127.9.238.253@KRBTEST.COM (short username oryx)
I20260926 16:44:11.516438 16811 webserver.cc:460] Webserver started at http://127.9.238.253:36621/ using document root <none> and password file <none>
I20260926 16:44:11.517334 16811 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:11.517480 16811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:11.517840 16811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:11.521063 16811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-1/data/instance:
uuid: "8d007fde43b449489bda0be2c54c1587"
format_stamp: "Formatted at 2026-09-26 16:44:11 on dist-test-slave-wt2j"
server_key: "65c5cfb54228f702a7ca2b45c70a9c7a"
server_key_iv: "8727f8d5c5996fe093ce6eb886604c87"
server_key_version: "encryptionkey@0"
I20260926 16:44:11.521907 16811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-1/wal/instance:
uuid: "8d007fde43b449489bda0be2c54c1587"
format_stamp: "Formatted at 2026-09-26 16:44:11 on dist-test-slave-wt2j"
server_key: "65c5cfb54228f702a7ca2b45c70a9c7a"
server_key_iv: "8727f8d5c5996fe093ce6eb886604c87"
server_key_version: "encryptionkey@0"
I20260926 16:44:11.528263 16811 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260926 16:44:11.532547 16826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:11.534736 16811 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260926 16:44:11.534929 16811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-1/wal
uuid: "8d007fde43b449489bda0be2c54c1587"
format_stamp: "Formatted at 2026-09-26 16:44:11 on dist-test-slave-wt2j"
server_key: "65c5cfb54228f702a7ca2b45c70a9c7a"
server_key_iv: "8727f8d5c5996fe093ce6eb886604c87"
server_key_version: "encryptionkey@0"
I20260926 16:44:11.535168 16811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:11.555254 16811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:11.560920 16811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:11.561193 16811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:11.583988 16811 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.253:38171
I20260926 16:44:11.584051 16877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.253:38171 every 8 connection(s)
I20260926 16:44:11.585682 16811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-1/data/info.pb
I20260926 16:44:11.590121 16878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:11.593461 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 16811
I20260926 16:44:11.597605 16878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:44:11 dist-test-slave-wt2j krb5kdc[16303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.238.253@KRBTEST.COM
Sep 26 16:44:11 dist-test-slave-wt2j krb5kdc[16303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441051, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.253@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM
I20260926 16:44:11.621865 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:11.602770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38980 (local address 127.9.238.254:40511)
0926 16:44:11.603329 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:11.603337 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:11.604019 (+ 682us) server_negotiation.cc:407] Connection header received
0926 16:44:11.604268 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:11.604273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:11.604355 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:11.604472 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:11.606022 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.607285 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.609533 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.610090 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.610208 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:11.614894 (+ 4686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:11.614904 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:11.614909 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:11.614969 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:11.618528 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:11.619412 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:11.619418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:11.619422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:11.619632 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:11.620051 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:11.620057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:11.620060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:11.620436 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:11.620604 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:11.621161 (+ 557us) server_negotiation.cc:299] Negotiation successful
0926 16:44:11.621451 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":302,"threads_started":1}
I20260926 16:44:11.623027 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:11.596224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52080 (local address 127.9.238.253:38171)
0926 16:44:11.597180 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:11.597195 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:11.597254 (+ 59us) server_negotiation.cc:407] Connection header received
0926 16:44:11.599092 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:11.599118 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:11.599714 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:11.600194 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:11.602214 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.604509 (+ 2295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.606292 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.607493 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.607743 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:11.613143 (+ 5400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:11.613158 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:11.613182 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:11.613239 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:11.617838 (+ 4599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:11.618820 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:11.618832 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:11.618835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:11.619092 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:11.619493 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:11.619500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:11.619508 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:11.620195 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:11.620537 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:11.620972 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 16:44:11.621518 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":438,"threads_started":1}
I20260926 16:44:11.623059 16883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:11.602899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.0.0.1:38980)
0926 16:44:11.603866 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:11.603912 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:11.604131 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:11.604558 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:11.604565 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:11.605441 (+ 876us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:11.605782 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:11.605795 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.607482 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.607486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:11.609343 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:11.609362 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.610360 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.610368 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:11.610571 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:11.611406 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:11.611431 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:11.614623 (+ 3192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:11.618700 (+ 4077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:11.618714 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:11.618740 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:11.619263 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:11.619759 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:11.619764 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:11.619768 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:11.619926 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:11.620574 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:11.620584 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:11.620836 (+ 252us) client_negotiation.cc:769] Sending connection context
0926 16:44:11.621055 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 16:44:11.621741 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":170,"thread_start_us":504,"threads_started":1}
I20260926 16:44:11.625607 16878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:44:11 dist-test-slave-wt2j krb5kdc[16303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441051, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.253@KRBTEST.COM for oryx/127.9.238.253@KRBTEST.COM
I20260926 16:44:11.642006 16883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:11.626408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:38171 (local address 127.0.0.1:52086)
0926 16:44:11.626666 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:11.626692 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:11.626814 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:11.627295 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:11.627299 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:11.627816 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:11.628002 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:11.628010 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.629093 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.629097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:11.630167 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:11.630178 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.630928 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.630932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:11.631092 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:11.632143 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:11.632164 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:11.634867 (+ 2703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:11.638472 (+ 3605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:11.638477 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:11.638481 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:11.639119 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:11.639765 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:11.639769 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:11.639771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:11.639927 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:11.640596 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:11.640600 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:11.640809 (+ 209us) client_negotiation.cc:769] Sending connection context
0926 16:44:11.640963 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 16:44:11.641572 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 16:44:11.642069 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:11.626496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52086 (local address 127.9.238.253:38171)
0926 16:44:11.626716 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:11.626723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:11.626747 (+ 24us) server_negotiation.cc:407] Connection header received
0926 16:44:11.626966 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:11.626971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:11.627061 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:11.627168 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:11.628171 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.628944 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.630315 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.630762 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.630869 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:11.635162 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:11.635168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:11.635172 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:11.635212 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:11.638256 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:11.639289 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:11.639294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:11.639297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:11.639532 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:11.640087 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:11.640091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:11.640094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:11.640420 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:11.640553 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:11.641102 (+ 549us) server_negotiation.cc:299] Negotiation successful
0926 16:44:11.641751 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":386,"server-negotiator.queue_time_us":82}
I20260926 16:44:11.647924 16878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587: Bootstrap starting.
I20260926 16:44:11.651389 16878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:11.652529 16878 log.cc:824] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:11.655768 16878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587: No bootstrap required, opened a new log
I20260926 16:44:11.663349 16878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } }
I20260926 16:44:11.663816 16878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:11.663936 16878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d007fde43b449489bda0be2c54c1587, State: Initialized, Role: FOLLOWER
I20260926 16:44:11.664619 16878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } }
I20260926 16:44:11.665753 16885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } } }
I20260926 16:44:11.666149 16885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:44:11.666756 16878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 16:44:11.678660 16896 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:44:11.678853 16896 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:44:11.685441 16811 master_runner.cc:428] Detected that this master 8d007fde43b449489bda0be2c54c1587 is joining an existing cluster
I20260926 16:44:11.685596 16811 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.238.253:38171
WARNING: no policy specified for oryx/127.9.238.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.238.252@KRBTEST.COM" created.
I20260926 16:44:11.712729 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:11.690652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38994 (local address 127.9.238.254:40511)
0926 16:44:11.690932 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:11.690942 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:11.691796 (+ 854us) server_negotiation.cc:407] Connection header received
0926 16:44:11.692105 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:11.692114 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:11.692247 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:11.692330 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:11.694110 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.695333 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.698835 (+ 3502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.699743 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.699857 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:11.704311 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:11.704319 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:11.704323 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:11.704373 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:11.708605 (+ 4232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:11.709684 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:11.709691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:11.709696 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:11.709901 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:11.710495 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:11.710501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:11.710505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:11.710970 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:11.711265 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:11.712027 (+ 762us) server_negotiation.cc:299] Negotiation successful
0926 16:44:11.712309 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 16:44:11.714404 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:11.690651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.0.0.1:38994)
0926 16:44:11.691662 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:11.691687 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:11.691940 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:11.692589 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:11.692597 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:11.693424 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:11.693905 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:11.693926 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.695542 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.695571 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:11.697481 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:11.698657 (+ 1176us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.700213 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.700221 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:11.700478 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:11.701371 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:11.701400 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:11.704061 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:11.708814 (+ 4753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:11.708827 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:11.708833 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:11.709493 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:11.710043 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:11.710049 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:11.710052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:11.710294 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:11.711358 (+ 1064us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:11.711367 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:11.711660 (+ 293us) client_negotiation.cc:769] Sending connection context
0926 16:44:11.713445 (+ 1785us) client_negotiation.cc:240] Negotiation successful
0926 16:44:11.713878 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":580,"threads_started":1}
I20260926 16:44:11.720799 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:11.691809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:38171 (local address 127.0.0.1:52092)
0926 16:44:11.703704 (+ 11895us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:11.703727 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:11.703961 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:11.704450 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:11.704455 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:11.705133 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:11.705454 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:11.705469 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.707230 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.707236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:11.709142 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:11.709154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.710284 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.710290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:11.710482 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:11.711539 (+ 1057us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:11.711603 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:11.712816 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:11.717447 (+ 4631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:11.717452 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:11.717455 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:11.718054 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:11.718561 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:11.718565 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:11.718569 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:11.718755 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:11.719596 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:11.719604 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:11.719766 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 16:44:11.719913 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 16:44:11.720250 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11692,"thread_start_us":11317,"threads_started":1}
I20260926 16:44:11.721907 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:11.692241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52092 (local address 127.9.238.253:38171)
0926 16:44:11.692424 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:11.692430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:11.703844 (+ 11414us) server_negotiation.cc:407] Connection header received
0926 16:44:11.704122 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:11.704130 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:11.704198 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:11.704260 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:11.705604 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.706949 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.709286 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.710077 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.710230 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:11.713035 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:11.713041 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:11.713046 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:11.713093 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:11.717278 (+ 4185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:11.718196 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:11.718201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:11.718204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:11.718414 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:11.718872 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:11.718882 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:11.718889 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:11.719311 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:11.719480 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:11.721282 (+ 1802us) server_negotiation.cc:299] Negotiation successful
0926 16:44:11.721518 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 16:44:11 dist-test-slave-wt2j krb5kdc[16303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441048, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.254@KRBTEST.COM for oryx/127.9.238.253@KRBTEST.COM
I20260926 16:44:11.743511 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:11.722724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52096 (local address 127.9.238.253:38171)
0926 16:44:11.722962 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:11.722971 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:11.726496 (+ 3525us) server_negotiation.cc:407] Connection header received
0926 16:44:11.726934 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:11.726941 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:11.727044 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:11.727176 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:11.728592 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.729526 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.730987 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.731749 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.731892 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:11.736447 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:11.736453 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:11.736456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:11.736491 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:11.740112 (+ 3621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:11.741011 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:11.741017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:11.741021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:11.741168 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:11.741679 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:11.741684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:11.741686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:11.742050 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:11.742250 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:11.742832 (+ 582us) server_negotiation.cc:299] Negotiation successful
0926 16:44:11.743075 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 16:44:11.743604 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:11.722258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:38171 (local address 127.0.0.1:52096)
0926 16:44:11.726318 (+ 4060us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:11.726400 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:11.726661 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:11.727230 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:11.727238 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:11.728154 (+ 916us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:11.728394 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:11.728404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.729672 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.729681 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:11.730808 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:11.730820 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.731934 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.731942 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:11.732246 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:11.733297 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:11.733321 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:11.736198 (+ 2877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:11.740323 (+ 4125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:11.740346 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:11.740394 (+ 48us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:11.740866 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:11.741307 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:11.741312 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:11.741316 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:11.741553 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:11.742238 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:11.742248 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:11.742436 (+ 188us) client_negotiation.cc:769] Sending connection context
0926 16:44:11.742638 (+ 202us) client_negotiation.cc:240] Negotiation successful
0926 16:44:11.743089 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3790,"thread_start_us":3632,"threads_started":1}
I20260926 16:44:11.746443 16348 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.238.253:38171: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8d007fde43b449489bda0be2c54c1587"
member_type: NON_VOTER
last_known_addr {
host: "127.9.238.253"
port: 38171
}
attrs {
promote: true
}
}
dest_uuid: "212c6cc5c6e6456b919eb5680dc91f11"
cas_config_opid_index: -1
I20260926 16:44:11.747736 16348 consensus_queue.cc:237] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: NON_VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.238.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.252.keytab.
Entry for principal oryx/127.9.238.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.252.keytab.
I20260926 16:44:11.752794 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.252:34171
--webserver_interface=127.9.238.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.252.keytab
--principal=oryx/127.9.238.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:39619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:40511,127.9.238.253:38171,127.9.238.252:34171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:44:11.752909 16853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:11.757387 16853 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [term 1 FOLLOWER]: Refusing update from remote peer 212c6cc5c6e6456b919eb5680dc91f11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 16:44:11.768018 16909 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: NON_VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:44:11.765983 16908 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8d007fde43b449489bda0be2c54c1587 (127.9.238.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: NON_VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: true } } }
I20260926 16:44:11.772528 16908 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.238.253:38171
I20260926 16:44:11.773382 16811 master.cc:537] Master@127.9.238.253:38171 shutting down...
I20260926 16:44:11.773118 16909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "212c6cc5c6e6456b919eb5680dc91f11" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: NON_VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: true } } }
I20260926 16:44:11.774127 16909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:11.798651 16853 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8d007fde43b449489bda0be2c54c1587 (127.9.238.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: NON_VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: true } } }
I20260926 16:44:11.824427 16885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 212c6cc5c6e6456b919eb5680dc91f11. Latest consensus state: current_term: 1 leader_uuid: "212c6cc5c6e6456b919eb5680dc91f11" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: NON_VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: true } } }
I20260926 16:44:11.824707 16885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [sys.catalog]: This master's current role is: LEARNER
I20260926 16:44:11.824887 16915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "212c6cc5c6e6456b919eb5680dc91f11" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: NON_VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: true } } }
I20260926 16:44:11.825456 16915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [sys.catalog]: This master's current role is: LEARNER
I20260926 16:44:11.823750 16919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "212c6cc5c6e6456b919eb5680dc91f11" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: NON_VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: true } } }
I20260926 16:44:11.825855 16919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [sys.catalog]: This master's current role is: LEARNER
I20260926 16:44:11.828379 16811 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [term 1 LEARNER]: Raft consensus shutting down.
I20260926 16:44:11.829133 16811 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [term 1 LEARNER]: Raft consensus is shut down!
I20260926 16:44:11.829339 16811 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587: stopping tablet replica
I20260926 16:44:11.849690 16811 master.cc:552] Master@127.9.238.253:38171 shutdown complete.
I20260926 16:44:11.849820 16811 master_runner.cc:305] Clearing existing system tablet
I20260926 16:44:11.850334 16811 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:11.850463 16811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:11.856294 16811 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 16:44:11.858515 16921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:11.859045 16811 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:11.859381 16811 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 16:44:11.859481 16811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-1/wal
uuid: "8d007fde43b449489bda0be2c54c1587"
format_stamp: "Formatted at 2026-09-26 16:44:11 on dist-test-slave-wt2j"
server_key: "65c5cfb54228f702a7ca2b45c70a9c7a"
server_key_iv: "8727f8d5c5996fe093ce6eb886604c87"
server_key_version: "encryptionkey@0"
I20260926 16:44:11.865046 16811 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:44:11.873625 16811 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 16:44:11.873891 16811 log.cc:1196] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 16:44:11.874480 16811 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587: Deleting consensus metadata
I20260926 16:44:11.874778 16811 master_runner.cc:315] Copying system tablet from 127.9.238.254:40511
I20260926 16:44:11.881606 16811 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587: tablet copy: Beginning tablet copy session from remote peer at address 127.9.238.254:40511
I20260926 16:44:11.902026 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:11.882957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38998 (local address 127.9.238.254:40511)
0926 16:44:11.883226 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:11.883238 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:11.883978 (+ 740us) server_negotiation.cc:407] Connection header received
0926 16:44:11.884391 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:11.884400 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:11.884534 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:11.884700 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:11.886017 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.887345 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.889501 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.890274 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.890446 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:11.893643 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:11.893651 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:11.893656 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:11.893708 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:11.898344 (+ 4636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:11.899583 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:11.899593 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:11.899598 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:11.899794 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:11.900266 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:11.900273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:11.900277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:11.900668 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:11.900835 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:11.901390 (+ 555us) server_negotiation.cc:299] Negotiation successful
0926 16:44:11.901642 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260926 16:44:11.902174 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:11.882876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.0.0.1:38998)
0926 16:44:11.883808 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:11.883843 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:11.884134 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:11.884764 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:11.884769 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:11.885421 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:11.885807 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:11.885835 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.887539 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.887571 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:11.889270 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:11.889296 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:11.890501 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:11.890509 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:11.890708 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:11.891835 (+ 1127us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:11.891867 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:11.893293 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:11.898598 (+ 5305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:11.898607 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:11.898612 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:11.899328 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:11.899934 (+ 606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:11.899940 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:11.899944 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:11.900119 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:11.900814 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:11.900819 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:11.901057 (+ 238us) client_negotiation.cc:769] Sending connection context
0926 16:44:11.901231 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 16:44:11.901628 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"thread_start_us":574,"threads_started":1}
I20260926 16:44:11.903729 16368 tablet_copy_service.cc:140] P 212c6cc5c6e6456b919eb5680dc91f11: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8d007fde43b449489bda0be2c54c1587 ({username='oryx', principal='oryx/127.9.238.253@KRBTEST.COM'} at 127.0.0.1:38998)
I20260926 16:44:11.904156 16368 tablet_copy_service.cc:161] P 212c6cc5c6e6456b919eb5680dc91f11: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8d007fde43b449489bda0be2c54c1587 at {username='oryx', principal='oryx/127.9.238.253@KRBTEST.COM'} at 127.0.0.1:38998: session id = 8d007fde43b449489bda0be2c54c1587-00000000000000000000000000000000
I20260926 16:44:11.909823 16368 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 16:44:11.913317 16811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 16:44:11.921500 16811 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587: tablet copy: Starting download of 0 data blocks...
I20260926 16:44:11.921921 16811 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587: tablet copy: Starting download of 1 WAL segments...
I20260926 16:44:11.925896 16811 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 16:44:11.931792 16368 tablet_copy_service.cc:342] P 212c6cc5c6e6456b919eb5680dc91f11: Request end of tablet copy session 8d007fde43b449489bda0be2c54c1587-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.238.253@KRBTEST.COM'} at 127.0.0.1:38998
I20260926 16:44:11.932209 16368 tablet_copy_service.cc:433] P 212c6cc5c6e6456b919eb5680dc91f11: ending tablet copy session 8d007fde43b449489bda0be2c54c1587-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8d007fde43b449489bda0be2c54c1587
I20260926 16:44:11.947670 16811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:11.950984 16928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:11.952862 16929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:11.959872 16931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:11.965370 16811 server_base.cc:1034] running on GCE node
I20260926 16:44:11.965935 16811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:11.966881 16811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:11.968114 16811 hybrid_clock.cc:648] HybridClock initialized: now 1790441051968072 us; error 100 us; skew 500 ppm
I20260926 16:44:11.968380 16811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:11 dist-test-slave-wt2j krb5kdc[16303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441051, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:11.971131 16811 init.cc:377] Logged in from keytab as oryx/127.9.238.253@KRBTEST.COM (short username oryx)
I20260926 16:44:11.975683 16811 webserver.cc:460] Webserver started at http://127.9.238.253:40303/ using document root <none> and password file <none>
I20260926 16:44:11.976174 16811 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:11.976274 16811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:11.980837 16811 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 16:44:11.983489 16938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:11.984212 16811 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 16:44:11.984442 16811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-1/wal
uuid: "8d007fde43b449489bda0be2c54c1587"
format_stamp: "Formatted at 2026-09-26 16:44:11 on dist-test-slave-wt2j"
server_key: "65c5cfb54228f702a7ca2b45c70a9c7a"
server_key_iv: "8727f8d5c5996fe093ce6eb886604c87"
server_key_version: "encryptionkey@0"
I20260926 16:44:11.984694 16811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:12.003490 16811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:12.014837 16811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:12.030719 16805 heartbeater.cc:503] Master 127.9.238.254:40511 was elected leader, sending a full tablet report...
I20260926 16:44:12.092168 16811 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.253:38171
I20260926 16:44:12.093590 16811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-1/data/info.pb
I20260926 16:44:12.099658 16989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.253:38171 every 8 connection(s)
I20260926 16:44:12.113240 16990 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:44:12.117381 16990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587: Bootstrap starting.
I20260926 16:44:12.135077 16990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:44:12.135797 16990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587: Bootstrap complete.
I20260926 16:44:12.137102 16990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: NON_VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: true } }
I20260926 16:44:12.137497 16990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d007fde43b449489bda0be2c54c1587, State: Initialized, Role: LEARNER
I20260926 16:44:12.137811 16990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: NON_VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: true } }
I20260926 16:44:12.139213 16990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:44:12.138948 16993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: NON_VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: true } } }
I20260926 16:44:12.139604 16993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [sys.catalog]: This master's current role is: LEARNER
I20260926 16:44:12.150580 17004 catalog_manager.cc:1259] Loaded cluster ID: 58607067b23c4cbca0a242e80249d2d2
I20260926 16:44:12.150730 17004 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587: loading cluster ID for follower catalog manager: success
I20260926 16:44:12.154942 17004 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587: acquiring CA information for follower catalog manager: success
I20260926 16:44:12.158102 17004 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 16:44:12.207056 16911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:12.207373 16911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:12.207499 16911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:12.215155 16911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:44:12.215299 16911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:12.215346 16911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:12.215397 16911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:44:12.215443 16911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:44:12.226533 16911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:39619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:40511,127.9.238.253:38171,127.9.238.252:34171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.252.keytab
--principal=oryx/127.9.238.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.252:34171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.238.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:12.228197 16911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:12.229713 16911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:12.245978 17010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:12.246397 17012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:12.245958 17009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:12.248183 16911 server_base.cc:1034] running on GCE node
I20260926 16:44:12.249135 16911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:12.250950 16911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:12.252424 16911 hybrid_clock.cc:648] HybridClock initialized: now 1790441052252349 us; error 67 us; skew 500 ppm
I20260926 16:44:12.252843 16911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:12 dist-test-slave-wt2j krb5kdc[16303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441052, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:12.256211 16911 init.cc:377] Logged in from keytab as oryx/127.9.238.252@KRBTEST.COM (short username oryx)
I20260926 16:44:12.258025 16911 webserver.cc:460] Webserver started at http://127.9.238.252:45165/ using document root <none> and password file <none>
I20260926 16:44:12.258771 16911 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:12.258919 16911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:12.259240 16911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:12.262645 16908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "212c6cc5c6e6456b919eb5680dc91f11" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: NON_VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: true } } }
I20260926 16:44:12.262931 16908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:12.262732 16911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-2/data/instance:
uuid: "83503b55cdba4750ad2e659cdf5f85d6"
format_stamp: "Formatted at 2026-09-26 16:44:12 on dist-test-slave-wt2j"
server_key: "312e7e08e5d3237a589721b155bd35a4"
server_key_iv: "9ae527493c3f924a8afce7020ac5d26f"
server_key_version: "encryptionkey@0"
I20260926 16:44:12.263751 16911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-2/wal/instance:
uuid: "83503b55cdba4750ad2e659cdf5f85d6"
format_stamp: "Formatted at 2026-09-26 16:44:12 on dist-test-slave-wt2j"
server_key: "312e7e08e5d3237a589721b155bd35a4"
server_key_iv: "9ae527493c3f924a8afce7020ac5d26f"
server_key_version: "encryptionkey@0"
I20260926 16:44:12.271217 16911 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s
I20260926 16:44:12.272408 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:12.263081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:38171 (local address 127.0.0.1:52106)
0926 16:44:12.263732 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:12.263759 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:12.264031 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:12.265126 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:12.265132 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:12.265176 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:12.265439 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:12.265451 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.267744 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.267753 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:12.269651 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:12.269666 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.271311 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.271319 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:12.271626 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:12.271707 (+ 81us) client_negotiation.cc:769] Sending connection context
0926 16:44:12.271844 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 16:44:12.272028 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":777,"threads_started":1}
I20260926 16:44:12.272477 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:12.263656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52106 (local address 127.9.238.253:38171)
0926 16:44:12.264692 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:12.264698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:12.264725 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:44:12.264804 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:12.264809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:12.264899 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:12.265030 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:12.265650 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.267505 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.269809 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.271072 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.271230 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:12.271509 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:12.271895 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 16:44:12.272046 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":859,"thread_start_us":657,"threads_started":1}
I20260926 16:44:12.273667 16993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 212c6cc5c6e6456b919eb5680dc91f11. Latest consensus state: current_term: 1 leader_uuid: "212c6cc5c6e6456b919eb5680dc91f11" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: NON_VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: true } } }
I20260926 16:44:12.273954 16993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [sys.catalog]: This master's current role is: LEARNER
I20260926 16:44:12.274899 16910 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11: attempting to promote NON_VOTER 8d007fde43b449489bda0be2c54c1587 to VOTER
I20260926 16:44:12.275974 16910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } }
I20260926 16:44:12.277168 17021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:12.278666 16965 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [term 1 LEARNER]: Refusing update from remote peer 212c6cc5c6e6456b919eb5680dc91f11: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 16:44:12.279670 16911 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260926 16:44:12.279615 16908 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 16:44:12.279875 16911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-2/wal
uuid: "83503b55cdba4750ad2e659cdf5f85d6"
format_stamp: "Formatted at 2026-09-26 16:44:12 on dist-test-slave-wt2j"
server_key: "312e7e08e5d3237a589721b155bd35a4"
server_key_iv: "9ae527493c3f924a8afce7020ac5d26f"
server_key_version: "encryptionkey@0"
I20260926 16:44:12.280144 16911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:12.284180 16910 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8d007fde43b449489bda0be2c54c1587 (127.9.238.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } }
I20260926 16:44:12.284986 16965 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8d007fde43b449489bda0be2c54c1587 (127.9.238.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } }
I20260926 16:44:12.288900 16993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "212c6cc5c6e6456b919eb5680dc91f11" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } }
I20260926 16:44:12.289105 16993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:44:12.289819 16909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "212c6cc5c6e6456b919eb5680dc91f11" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } }
I20260926 16:44:12.290202 16909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:12.290759 16909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "212c6cc5c6e6456b919eb5680dc91f11" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } }
I20260926 16:44:12.291038 16909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:12.302935 16911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:12.308306 16911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:12.308650 16911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:12.332485 16911 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.252:34171
I20260926 16:44:12.332537 17073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.252:34171 every 8 connection(s)
I20260926 16:44:12.334352 16911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-2/data/info.pb
I20260926 16:44:12.338774 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 16911
I20260926 16:44:12.339012 17074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:12.350150 17074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:44:12 dist-test-slave-wt2j krb5kdc[16303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.238.252@KRBTEST.COM
Sep 26 16:44:12 dist-test-slave-wt2j krb5kdc[16303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441052, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.252@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM
I20260926 16:44:12.368796 17077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:12.341621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60394 (local address 127.9.238.252:34171)
0926 16:44:12.342927 (+ 1306us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:12.342945 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:12.342999 (+ 54us) server_negotiation.cc:407] Connection header received
0926 16:44:12.344879 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:12.344908 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:12.345498 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:12.345954 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:12.347975 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.349683 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.351692 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.352592 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.352844 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:12.358599 (+ 5755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:12.358614 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:12.358639 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:12.358700 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:12.362924 (+ 4224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:12.363923 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:12.363934 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:12.363946 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:12.364195 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:12.364707 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:12.364712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:12.364715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:12.365525 (+ 810us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:12.365916 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:12.366685 (+ 769us) server_negotiation.cc:299] Negotiation successful
0926 16:44:12.367193 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1036,"thread_start_us":734,"threads_started":1}
I20260926 16:44:12.372069 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:12.353530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39008 (local address 127.9.238.254:40511)
0926 16:44:12.353806 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:12.353812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:12.354721 (+ 909us) server_negotiation.cc:407] Connection header received
0926 16:44:12.354974 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:12.354979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:12.355066 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:12.355155 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:12.356866 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.357899 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.360334 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.360821 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.360988 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:12.365222 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:12.365228 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:12.365233 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:12.365272 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:12.369205 (+ 3933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:12.370022 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:12.370027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:12.370031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:12.370165 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:12.370539 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:12.370544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:12.370548 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:12.370825 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:12.370976 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:12.371460 (+ 484us) server_negotiation.cc:299] Negotiation successful
0926 16:44:12.371703 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260926 16:44:12.372169 17078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:12.353393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.0.0.1:39008)
0926 16:44:12.354505 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:12.354573 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:12.354791 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:12.355278 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:12.355288 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:12.356284 (+ 996us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:12.356682 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:12.356694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.358109 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.358115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:12.360137 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:12.360154 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.361042 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.361049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:12.361272 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:12.362057 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:12.362082 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:12.364996 (+ 2914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:12.369370 (+ 4374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:12.369383 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:12.369390 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:12.369875 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:12.370276 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:12.370282 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:12.370285 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:12.370424 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:12.370955 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:12.370963 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:12.371137 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 16:44:12.371362 (+ 225us) client_negotiation.cc:240] Negotiation successful
0926 16:44:12.371733 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"thread_start_us":572,"threads_started":1}
I20260926 16:44:12.374596 17074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:44:12 dist-test-slave-wt2j krb5kdc[16303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441052, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.252@KRBTEST.COM for oryx/127.9.238.253@KRBTEST.COM
I20260926 16:44:12.388738 17078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:12.375519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:38171 (local address 127.0.0.1:52122)
0926 16:44:12.375758 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:12.375780 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:12.375867 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:12.376383 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:12.376387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:12.376741 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:12.376876 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:12.376884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.377866 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.377871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:12.378780 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:12.378791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.379582 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.379590 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:12.379773 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:12.380461 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:12.380479 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:12.382742 (+ 2263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:12.386122 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:12.386127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:12.386132 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:12.386599 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:12.387016 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:12.387023 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:12.387026 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:12.387174 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:12.387733 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:12.387737 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:12.387882 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 16:44:12.388026 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 16:44:12.388376 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260926 16:44:12.388738 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:12.375890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52122 (local address 127.9.238.253:38171)
0926 16:44:12.376122 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:12.376128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:12.376143 (+ 15us) server_negotiation.cc:407] Connection header received
0926 16:44:12.376189 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:12.376193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:12.376253 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:12.376363 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:12.377023 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.377730 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.378920 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.379364 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.379520 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:12.383012 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:12.383018 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:12.383022 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:12.383053 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:12.385969 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:12.386747 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:12.386754 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:12.386757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:12.386903 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:12.387283 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:12.387290 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:12.387293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:12.387611 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:12.387756 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:12.388113 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 16:44:12.388376 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 16:44:12.389858 17074 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.252" port: 34171 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:44:12 dist-test-slave-wt2j krb5kdc[16303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441052, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.252@KRBTEST.COM for oryx/127.9.238.252@KRBTEST.COM
I20260926 16:44:12.405361 17078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:12.390923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:34171 (local address 127.0.0.1:60410)
0926 16:44:12.391175 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:12.391266 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:12.391457 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:12.391848 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:12.391853 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:12.392341 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:12.392517 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:12.392527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.393601 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.393610 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:12.394548 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:12.394560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.395353 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.395360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:12.395662 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:12.396444 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:12.396465 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:12.398921 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:12.402492 (+ 3571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:12.402497 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:12.402501 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:12.402966 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:12.403397 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:12.403400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:12.403403 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:12.403661 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:12.404335 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:12.404340 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:12.404522 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 16:44:12.404662 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 16:44:12.404957 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 16:44:12.405766 17077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:12.390923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60410 (local address 127.9.238.252:34171)
0926 16:44:12.391175 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:12.391183 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:12.391363 (+ 180us) server_negotiation.cc:407] Connection header received
0926 16:44:12.391629 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:12.391637 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:12.391712 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:12.391833 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:12.392670 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.393453 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.394716 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.395157 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.395316 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:12.399221 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:12.399229 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:12.399233 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:12.399281 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:12.402298 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:12.403133 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:12.403138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:12.403140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:12.403273 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:12.403799 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:12.403803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:12.403806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:12.404168 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:12.404304 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:12.404765 (+ 461us) server_negotiation.cc:299] Negotiation successful
0926 16:44:12.405166 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":127,"server-negotiator.queue_time_us":93}
I20260926 16:44:12.411840 17074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6: Bootstrap starting.
I20260926 16:44:12.415881 17074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:12.417230 17074 log.cc:824] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:12.420701 17074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6: No bootstrap required, opened a new log
I20260926 16:44:12.428404 17074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 34171 } }
I20260926 16:44:12.428733 17074 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:12.428869 17074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83503b55cdba4750ad2e659cdf5f85d6, State: Initialized, Role: FOLLOWER
I20260926 16:44:12.429522 17074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 34171 } }
I20260926 16:44:12.430540 17080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 34171 } } }
I20260926 16:44:12.431010 17080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:44:12.431684 17074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 16:44:12.442016 17091 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:44:12.442216 17091 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:44:12.448768 16911 master_runner.cc:428] Detected that this master 83503b55cdba4750ad2e659cdf5f85d6 is joining an existing cluster
I20260926 16:44:12.448915 16911 master_runner.cc:432] Initiating AddMaster RPC to add 127.9.238.252:34171
I20260926 16:44:12.476266 17077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:12.454547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60422 (local address 127.9.238.252:34171)
0926 16:44:12.454808 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:12.454817 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:12.456214 (+ 1397us) server_negotiation.cc:407] Connection header received
0926 16:44:12.456412 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:12.456428 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:12.456498 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:12.456558 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:12.458506 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.459786 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.462287 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.463092 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.463198 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:12.466716 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:12.466722 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:12.466726 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:12.466761 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:12.472097 (+ 5336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:12.473186 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:12.473194 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:12.473197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:12.473373 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:12.473798 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:12.473803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:12.473806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:12.474422 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:12.474603 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:12.475539 (+ 936us) server_negotiation.cc:299] Negotiation successful
0926 16:44:12.475824 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":92}
I20260926 16:44:12.476876 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 16409
I20260926 16:44:12.477532 17097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:12.454596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:34171 (local address 127.0.0.1:60422)
0926 16:44:12.456128 (+ 1532us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:12.456144 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:12.456297 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:12.456770 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:12.456776 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:12.457748 (+ 972us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:12.458362 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:12.458370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.460035 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.460046 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:12.462110 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:12.462126 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.463597 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.463606 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:12.463903 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:12.464872 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:12.464888 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:12.466524 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:12.472199 (+ 5675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:12.472211 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:12.472218 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:12.473045 (+ 827us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:12.473517 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:12.473522 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:12.473529 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:12.473685 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:12.474695 (+ 1010us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:12.474701 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:12.475206 (+ 505us) client_negotiation.cc:769] Sending connection context
0926 16:44:12.476654 (+ 1448us) client_negotiation.cc:240] Negotiation successful
0926 16:44:12.477020 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1396,"mutex_wait_us":103,"spinlock_wait_cycles":576128,"thread_start_us":547,"threads_started":1}
I20260926 16:44:12.477538 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:12.453980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39016 (local address 127.9.238.254:40511)
0926 16:44:12.454174 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:12.454181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:12.455736 (+ 1555us) server_negotiation.cc:407] Connection header received
0926 16:44:12.456112 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:12.456119 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:12.456227 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:12.456310 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:12.458525 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.459869 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.462436 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.463113 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.463197 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:12.466768 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:12.466774 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:12.466778 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:12.466822 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:12.471010 (+ 4188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:12.473227 (+ 2217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:12.473235 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:12.473241 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:12.473436 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:12.473934 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:12.473940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:12.473944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:12.474474 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:12.474656 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:12.476863 (+ 2207us) server_negotiation.cc:299] Negotiation successful
0926 16:44:12.477176 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 16:44:12.478876 17098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:12.454395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.0.0.1:39016)
0926 16:44:12.455586 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:12.455608 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:12.455917 (+ 309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:12.456540 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:12.456546 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:12.457684 (+ 1138us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:12.458345 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:12.458359 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.460365 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.460377 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:12.462287 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:12.462298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.463531 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.463542 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:12.463780 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:12.464793 (+ 1013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:12.464832 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:12.466566 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:12.472273 (+ 5707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:12.472282 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:12.472286 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:12.473080 (+ 794us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:12.473596 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:12.473601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:12.473604 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:12.473798 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:12.474716 (+ 918us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:12.474720 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:12.475293 (+ 573us) client_negotiation.cc:769] Sending connection context
0926 16:44:12.478009 (+ 2716us) client_negotiation.cc:240] Negotiation successful
0926 16:44:12.478384 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"mutex_wait_us":65,"spinlock_wait_cycles":132096,"thread_start_us":1367,"threads_started":1}
I20260926 16:44:12.479338 17099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:12.456306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:38171 (local address 127.0.0.1:52132)
0926 16:44:12.458256 (+ 1950us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:12.458276 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:12.458535 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:12.459137 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:12.459142 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:12.459959 (+ 817us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:12.460614 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:12.460630 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.462116 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.462121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:12.464009 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:12.464023 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.465292 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.465299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:12.465537 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:12.466406 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:12.466431 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:12.467761 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:12.472133 (+ 4372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:12.472138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:12.472143 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:12.472967 (+ 824us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:12.473558 (+ 591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:12.473563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:12.473567 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:12.473756 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:12.474606 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:12.474611 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:12.475457 (+ 846us) client_negotiation.cc:769] Sending connection context
0926 16:44:12.475721 (+ 264us) client_negotiation.cc:240] Negotiation successful
0926 16:44:12.476079 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1702,"mutex_wait_us":7,"spinlock_wait_cycles":1322496,"thread_start_us":1149,"threads_started":1}
I20260926 16:44:12.481734 17019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:12.456804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52132 (local address 127.9.238.253:38171)
0926 16:44:12.456989 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:12.456995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:12.458395 (+ 1400us) server_negotiation.cc:407] Connection header received
0926 16:44:12.458698 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:12.458703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:12.458790 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:12.458870 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:12.460815 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.461916 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.464190 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.464867 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.464968 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:12.468007 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:12.468017 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:12.468024 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:12.468081 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:12.471958 (+ 3877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:12.473152 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:12.473169 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:12.473173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:12.473368 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:12.473969 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:12.473975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:12.473979 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:12.474360 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:12.474524 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:12.481058 (+ 6534us) server_negotiation.cc:299] Negotiation successful
0926 16:44:12.481348 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 16:44:12 dist-test-slave-wt2j krb5kdc[16303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441048, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.254@KRBTEST.COM for oryx/127.9.238.252@KRBTEST.COM
I20260926 16:44:12.503235 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:12.485657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:34171 (local address 127.0.0.1:60426)
0926 16:44:12.485918 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:12.485947 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:12.486135 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:12.486843 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:12.486849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:12.487416 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:12.487669 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:12.487685 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.488916 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.488925 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:12.489948 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:12.489959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.490906 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.490913 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:12.491105 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:12.491934 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:12.491960 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:12.494973 (+ 3013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:12.499792 (+ 4819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:12.499799 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:12.499805 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:12.500474 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:12.501014 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:12.501020 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:12.501024 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:12.501196 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:12.501987 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:12.501992 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:12.502207 (+ 215us) client_negotiation.cc:769] Sending connection context
0926 16:44:12.502384 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 16:44:12.502757 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 16:44:12.503253 17077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:12.485686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60426 (local address 127.9.238.252:34171)
0926 16:44:12.485939 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:12.485947 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:12.486049 (+ 102us) server_negotiation.cc:407] Connection header received
0926 16:44:12.486499 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:12.486506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:12.486638 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:12.486779 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:12.487853 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.488761 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.490109 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.490739 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.490893 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:12.495270 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:12.495276 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:12.495280 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:12.495321 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:12.499603 (+ 4282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:12.500678 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:12.500685 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:12.500688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:12.500852 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:12.501322 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:12.501331 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:12.501335 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:12.501852 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:12.502033 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:12.502476 (+ 443us) server_negotiation.cc:299] Negotiation successful
0926 16:44:12.502756 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 16:44:12.505034 16347 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.9.238.252:34171: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6"
member_type: NON_VOTER
last_known_addr {
host: "127.9.238.252"
port: 34171
}
attrs {
promote: true
}
}
dest_uuid: "212c6cc5c6e6456b919eb5680dc91f11"
cas_config_opid_index: 6
I20260926 16:44:12.505996 16347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: NON_VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: true } }
Sep 26 16:44:12 dist-test-slave-wt2j krb5kdc[16303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.238.193@KRBTEST.COM
I20260926 16:44:12.508523 16965 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [term 1 FOLLOWER]: Refusing update from remote peer 212c6cc5c6e6456b919eb5680dc91f11: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 16:44:12.509181 17049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:12.512120 16910 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 16:44:12.515457 17100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:12.478898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44250 (local address 127.9.238.193:40747)
0926 16:44:12.492298 (+ 13400us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:12.492308 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:12.492343 (+ 35us) server_negotiation.cc:407] Connection header received
0926 16:44:12.492425 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:12.492434 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:12.492760 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:12.492955 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:12.495197 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.496892 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.499347 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.500579 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.500743 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:12.507713 (+ 6970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:12.507739 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:12.507780 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:12.507825 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:12.511463 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:12.512510 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:12.512521 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:12.512525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:12.512746 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:12.513233 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:12.513240 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:12.513245 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:12.513681 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:12.514035 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:12.514586 (+ 551us) server_negotiation.cc:299] Negotiation successful
0926 16:44:12.514956 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13193,"thread_start_us":14256,"threads_started":1}
I20260926 16:44:12.515303 16910 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 83503b55cdba4750ad2e659cdf5f85d6 (127.9.238.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: NON_VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: true } } }
I20260926 16:44:12.518986 16965 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 83503b55cdba4750ad2e659cdf5f85d6 (127.9.238.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: NON_VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: true } } }
I20260926 16:44:12.520352 16532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:12.522704 17049 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [term 1 FOLLOWER]: Refusing update from remote peer 212c6cc5c6e6456b919eb5680dc91f11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 16:44:12.524197 16908 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: NON_VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:44:12.524260 16993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "212c6cc5c6e6456b919eb5680dc91f11" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: NON_VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: true } } }
I20260926 16:44:12.524531 16993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:44:12.531904 16910 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.9.238.252:34171
I20260926 16:44:12.532792 16911 master.cc:537] Master@127.9.238.252:34171 shutting down...
I20260926 16:44:12.532898 16909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "212c6cc5c6e6456b919eb5680dc91f11" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: NON_VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: true } } }
I20260926 16:44:12.533218 16909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:12.533571 16909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "212c6cc5c6e6456b919eb5680dc91f11" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: NON_VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: true } } }
I20260926 16:44:12.533847 16909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:12.547942 17049 raft_consensus.cc:858] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 34171 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: NON_VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } }
W20260926 16:44:12.564024 17028 tablet.cc:2369] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:44:12.548166 17049 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 34171 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: NON_VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } }
I20260926 16:44:12.585513 17049 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8d007fde43b449489bda0be2c54c1587 (127.9.238.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: NON_VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: true } } }
I20260926 16:44:12.590739 17108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "212c6cc5c6e6456b919eb5680dc91f11" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: NON_VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: true } } }
I20260926 16:44:12.591310 17108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 16:44:12.591171 17080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 212c6cc5c6e6456b919eb5680dc91f11. Latest consensus state: current_term: 1 leader_uuid: "212c6cc5c6e6456b919eb5680dc91f11" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: NON_VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: true } } }
I20260926 16:44:12.591512 17080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 16:44:12.591886 17105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "212c6cc5c6e6456b919eb5680dc91f11" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: NON_VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: true } } }
I20260926 16:44:12.592294 17105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 16:44:12.594288 16908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "212c6cc5c6e6456b919eb5680dc91f11" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: NON_VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: true } } }
I20260926 16:44:12.594578 16908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [sys.catalog]: This master's current role is: LEADER
W20260926 16:44:12.595877 16332 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260926 16:44:12.596006 16911 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260926 16:44:12.596516 16911 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260926 16:44:12.596680 16911 tablet_replica.cc:330] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6: stopping tablet replica
W20260926 16:44:12.602837 16332 consensus_peers.cc:487] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 -> Peer 83503b55cdba4750ad2e659cdf5f85d6 (127.9.238.252:34171): Couldn't send request to peer 83503b55cdba4750ad2e659cdf5f85d6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 16:44:12.616443 16911 master.cc:552] Master@127.9.238.252:34171 shutdown complete.
I20260926 16:44:12.616588 16911 master_runner.cc:305] Clearing existing system tablet
I20260926 16:44:12.617136 16911 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:12.617282 16911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:12.621388 16911 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 16:44:12.623652 17111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:12.624224 16911 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:12.624557 16911 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 16:44:12.624713 16911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-2/wal
uuid: "83503b55cdba4750ad2e659cdf5f85d6"
format_stamp: "Formatted at 2026-09-26 16:44:12 on dist-test-slave-wt2j"
server_key: "312e7e08e5d3237a589721b155bd35a4"
server_key_iv: "9ae527493c3f924a8afce7020ac5d26f"
server_key_version: "encryptionkey@0"
I20260926 16:44:12.631027 16911 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:44:12.641414 16911 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 16:44:12.641690 16911 log.cc:1196] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 16:44:12.642445 16911 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6: Deleting consensus metadata
I20260926 16:44:12.642760 16911 master_runner.cc:315] Copying system tablet from 127.9.238.254:40511
I20260926 16:44:12.651352 16911 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6: tablet copy: Beginning tablet copy session from remote peer at address 127.9.238.254:40511
I20260926 16:44:12.660507 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 16409
I20260926 16:44:12.671406 17117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:12.652507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.0.0.1:39026)
0926 16:44:12.653502 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:12.653532 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:12.653763 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:12.654280 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:12.654286 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:12.654989 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:12.655329 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:12.655343 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.656829 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.656837 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:12.659016 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:12.659033 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.660066 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.660073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:12.660293 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:12.661690 (+ 1397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:12.661718 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:12.663141 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:12.667844 (+ 4703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:12.667859 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:12.667865 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:12.668505 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:12.669036 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:12.669042 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:12.669046 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:12.669248 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:12.669979 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:12.669985 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:12.670246 (+ 261us) client_negotiation.cc:769] Sending connection context
0926 16:44:12.670454 (+ 208us) client_negotiation.cc:240] Negotiation successful
0926 16:44:12.670851 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"thread_start_us":643,"threads_started":1}
I20260926 16:44:12.671464 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:12.652677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39026 (local address 127.9.238.254:40511)
0926 16:44:12.652988 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:12.652997 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:12.653696 (+ 699us) server_negotiation.cc:407] Connection header received
0926 16:44:12.653950 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:12.653957 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:12.654072 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:12.654209 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:12.655531 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.656627 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.659206 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:12.659854 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:12.659992 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:12.663401 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:12.663410 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:12.663418 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:12.663471 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:12.667619 (+ 4148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:12.668693 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:12.668702 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:12.668709 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:12.668885 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:12.669382 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:12.669390 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:12.669396 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:12.669822 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:12.669979 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:12.670600 (+ 621us) server_negotiation.cc:299] Negotiation successful
0926 16:44:12.670883 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260926 16:44:12.672721 16368 tablet_copy_service.cc:140] P 212c6cc5c6e6456b919eb5680dc91f11: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 83503b55cdba4750ad2e659cdf5f85d6 ({username='oryx', principal='oryx/127.9.238.252@KRBTEST.COM'} at 127.0.0.1:39026)
I20260926 16:44:12.672911 16368 tablet_copy_service.cc:161] P 212c6cc5c6e6456b919eb5680dc91f11: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 83503b55cdba4750ad2e659cdf5f85d6 at {username='oryx', principal='oryx/127.9.238.252@KRBTEST.COM'} at 127.0.0.1:39026: session id = 83503b55cdba4750ad2e659cdf5f85d6-00000000000000000000000000000000
I20260926 16:44:12.677376 16368 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 16:44:12.680410 16911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 16:44:12.682385 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:40747
--local_ip_for_outbound_sockets=127.9.238.193
--tserver_master_addrs=127.9.238.254:40511
--webserver_port=41751
--webserver_interface=127.9.238.193
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.193.keytab
--principal=oryx/127.9.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:39619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.238.254:40511,127.9.238.253:38171
--tserver_master_addrs=127.9.238.254:40511,127.9.238.253:38171,127.9.238.252:34171 with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:44:12.689358 16911 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6: tablet copy: Starting download of 0 data blocks...
I20260926 16:44:12.689770 16911 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6: tablet copy: Starting download of 1 WAL segments...
I20260926 16:44:12.693310 16911 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 16:44:12.699246 16368 tablet_copy_service.cc:342] P 212c6cc5c6e6456b919eb5680dc91f11: Request end of tablet copy session 83503b55cdba4750ad2e659cdf5f85d6-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.9.238.252@KRBTEST.COM'} at 127.0.0.1:39026
I20260926 16:44:12.699452 16368 tablet_copy_service.cc:433] P 212c6cc5c6e6456b919eb5680dc91f11: ending tablet copy session 83503b55cdba4750ad2e659cdf5f85d6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 83503b55cdba4750ad2e659cdf5f85d6
I20260926 16:44:12.715440 16911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:12.719646 17119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:12.720161 17120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:12.728024 16911 server_base.cc:1034] running on GCE node
W20260926 16:44:12.728459 17122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:12.729250 16911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:12.730082 16911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:12.731319 16911 hybrid_clock.cc:648] HybridClock initialized: now 1790441052731242 us; error 57 us; skew 500 ppm
I20260926 16:44:12.731673 16911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:12 dist-test-slave-wt2j krb5kdc[16303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441052, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:12.734351 16911 init.cc:377] Logged in from keytab as oryx/127.9.238.252@KRBTEST.COM (short username oryx)
I20260926 16:44:12.740209 16911 webserver.cc:460] Webserver started at http://127.9.238.252:35105/ using document root <none> and password file <none>
I20260926 16:44:12.740661 16911 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:12.740779 16911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:12.748497 16911 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.003s
I20260926 16:44:12.752069 17129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:12.752715 16911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 16:44:12.752887 16911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-2/wal
uuid: "83503b55cdba4750ad2e659cdf5f85d6"
format_stamp: "Formatted at 2026-09-26 16:44:12 on dist-test-slave-wt2j"
server_key: "312e7e08e5d3237a589721b155bd35a4"
server_key_iv: "9ae527493c3f924a8afce7020ac5d26f"
server_key_version: "encryptionkey@0"
I20260926 16:44:12.753147 16911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:12.773811 16911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:12.778851 16911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:12.819507 16911 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.252:34171
I20260926 16:44:12.819821 17180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.252:34171 every 8 connection(s)
I20260926 16:44:12.820854 16911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/master-2/data/info.pb
I20260926 16:44:12.827129 17181 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:44:12.832580 17181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6: Bootstrap starting.
I20260926 16:44:12.850225 17181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:44:12.850914 17181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6: Bootstrap complete.
I20260926 16:44:12.852159 17181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: NON_VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: true } }
I20260926 16:44:12.852511 17181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83503b55cdba4750ad2e659cdf5f85d6, State: Initialized, Role: LEARNER
I20260926 16:44:12.852883 17181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: NON_VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: true } }
I20260926 16:44:12.854226 17181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:44:12.853915 17184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: NON_VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: true } } }
I20260926 16:44:12.854668 17184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [sys.catalog]: This master's current role is: LEARNER
I20260926 16:44:12.869023 17195 catalog_manager.cc:1259] Loaded cluster ID: 58607067b23c4cbca0a242e80249d2d2
I20260926 16:44:12.869166 17195 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6: loading cluster ID for follower catalog manager: success
I20260926 16:44:12.873394 17195 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6: acquiring CA information for follower catalog manager: success
I20260926 16:44:12.876446 17195 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 16:44:13.066897 17118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:13.067190 17118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:13.067332 17118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:13.074879 17118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:13.075012 17118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:13.075127 17118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:44:13.085743 17118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:39619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.193.keytab
--principal=oryx/127.9.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:40747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=41751
--tserver_master_addrs=127.9.238.254:40511,127.9.238.253:38171,127.9.238.252:34171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:13.087237 17118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:13.088688 17118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:13.104395 17200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:13.104290 17203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:13.104378 17201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:13.106491 17118 server_base.cc:1034] running on GCE node
I20260926 16:44:13.107342 17118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:13.108740 17118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:13.109998 17118 hybrid_clock.cc:648] HybridClock initialized: now 1790441053109964 us; error 69 us; skew 500 ppm
I20260926 16:44:13.110427 17118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:13 dist-test-slave-wt2j krb5kdc[16303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441053, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:13.115185 17118 init.cc:377] Logged in from keytab as oryx/127.9.238.193@KRBTEST.COM (short username oryx)
I20260926 16:44:13.117056 17118 webserver.cc:460] Webserver started at http://127.9.238.193:41751/ using document root <none> and password file <none>
I20260926 16:44:13.117794 17118 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:13.117916 17118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:13.124238 17118 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:44:13.127858 17211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:13.129791 17118 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260926 16:44:13.130035 17118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "e7a44bfa35494003aadce77ec43974a8"
format_stamp: "Formatted at 2026-09-26 16:44:09 on dist-test-slave-wt2j"
server_key: "92b11cdd5b80c9d72d502792e4230658"
server_key_iv: "2f874528f41b89a587151517f37b5af5"
server_key_version: "encryptionkey@0"
I20260926 16:44:13.131140 17118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:13.160678 17118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:13.165364 17118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:13.165688 17118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:13.166883 17118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:13.169655 17118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:13.169752 17118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:13.169916 17118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:13.169996 17118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:13.172868 17206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "212c6cc5c6e6456b919eb5680dc91f11" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: NON_VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: true } } }
I20260926 16:44:13.173225 17206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:13.194556 17224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:13.173267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60440 (local address 127.9.238.252:34171)
0926 16:44:13.187436 (+ 14169us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:13.187441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:13.187463 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:44:13.187541 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:13.187605 (+ 64us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:13.187718 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:13.188481 (+ 763us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:13.188743 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.190208 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.192462 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.193358 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.193517 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:13.193818 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:13.193981 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 16:44:13.194124 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14000,"thread_start_us":13867,"threads_started":1}
I20260926 16:44:13.194556 17222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:13.173102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:34171 (local address 127.0.0.1:60440)
0926 16:44:13.185625 (+ 12523us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:13.185656 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:13.185889 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:13.187967 (+ 2078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:13.187975 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:13.188012 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:13.188328 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:13.188341 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.190362 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.190371 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:13.192303 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:13.192319 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.193569 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.193575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:13.193771 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:13.193825 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 16:44:13.193960 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 16:44:13.194125 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12350,"thread_start_us":12895,"threads_started":1}
I20260926 16:44:13.195785 17156 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260926 16:44:13.196318 17184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 212c6cc5c6e6456b919eb5680dc91f11. Latest consensus state: current_term: 1 leader_uuid: "212c6cc5c6e6456b919eb5680dc91f11" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: NON_VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: true } } }
I20260926 16:44:13.196619 17184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [sys.catalog]: This master's current role is: LEARNER
I20260926 16:44:13.197512 17219 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11: attempting to promote NON_VOTER 83503b55cdba4750ad2e659cdf5f85d6 to VOTER
I20260926 16:44:13.198279 17219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: false } }
I20260926 16:44:13.201052 17156 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [term 1 LEARNER]: Refusing update from remote peer 212c6cc5c6e6456b919eb5680dc91f11: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 16:44:13.201341 16965 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [term 1 FOLLOWER]: Refusing update from remote peer 212c6cc5c6e6456b919eb5680dc91f11: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260926 16:44:13.201903 17220 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 16:44:13.202397 17206 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 16:44:13.205613 17206 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 83503b55cdba4750ad2e659cdf5f85d6 (127.9.238.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: false } } }
I20260926 16:44:13.206349 16965 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 83503b55cdba4750ad2e659cdf5f85d6 (127.9.238.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: false } } }
I20260926 16:44:13.208716 17206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "212c6cc5c6e6456b919eb5680dc91f11" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: false } } }
I20260926 16:44:13.208977 17219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "212c6cc5c6e6456b919eb5680dc91f11" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: false } } }
I20260926 16:44:13.209201 17219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:13.209002 17206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:13.212404 17262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "212c6cc5c6e6456b919eb5680dc91f11" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: false } } }
I20260926 16:44:13.212878 17262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:44:13.212906 17219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "212c6cc5c6e6456b919eb5680dc91f11" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: false } } }
I20260926 16:44:13.213320 17219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:13.213689 17156 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 83503b55cdba4750ad2e659cdf5f85d6 (127.9.238.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: false } } }
I20260926 16:44:13.218654 17184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "212c6cc5c6e6456b919eb5680dc91f11" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: false } } }
I20260926 16:44:13.219086 17184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:44:13.239802 17118 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:40747
I20260926 16:44:13.239969 17330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:40747 every 8 connection(s)
I20260926 16:44:13.241436 17118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:44:13.252272 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 17118
I20260926 16:44:13.252688 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 16543
Sep 26 16:44:13 dist-test-slave-wt2j krb5kdc[16303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441053, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.252@KRBTEST.COM
Sep 26 16:44:13 dist-test-slave-wt2j krb5kdc[16303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441053, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.253@KRBTEST.COM
I20260926 16:44:13.278815 17224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:13.246697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:34943 (local address 127.9.238.252:34171)
0926 16:44:13.247310 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:13.247318 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:13.247536 (+ 218us) server_negotiation.cc:407] Connection header received
0926 16:44:13.251604 (+ 4068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:13.251614 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:13.251746 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:13.251849 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:13.256326 (+ 4477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.257838 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.261703 (+ 3865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.262572 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.262707 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:13.270459 (+ 7752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:13.270464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:13.270468 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:13.270506 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:13.274198 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:13.275449 (+ 1251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:13.275460 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:13.275465 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:13.275778 (+ 313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:13.276386 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:13.276393 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:13.276396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:13.276783 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:13.276906 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:13.278168 (+ 1262us) server_negotiation.cc:299] Negotiation successful
0926 16:44:13.278431 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435}
Sep 26 16:44:13 dist-test-slave-wt2j krb5kdc[16303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441053, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM
I20260926 16:44:13.281375 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:13.246016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:34171 (local address 127.9.238.193:34943)
0926 16:44:13.247289 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:13.247369 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:13.251040 (+ 3671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:13.252657 (+ 1617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:13.252678 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:13.255772 (+ 3094us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:13.256106 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:13.256122 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.258097 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.258104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:13.261458 (+ 3354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:13.261491 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.263154 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.263163 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:13.263651 (+ 488us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:13.265008 (+ 1357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:13.265046 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:13.270147 (+ 5101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:13.274528 (+ 4381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:13.274542 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:13.274563 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:13.275288 (+ 725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:13.275938 (+ 650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:13.275955 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:13.275958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:13.276236 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:13.277110 (+ 874us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:13.277120 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:13.277645 (+ 525us) client_negotiation.cc:769] Sending connection context
0926 16:44:13.279152 (+ 1507us) client_negotiation.cc:240] Negotiation successful
0926 16:44:13.279807 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"mutex_wait_us":56,"spinlock_wait_cycles":2304,"thread_start_us":634,"threads_started":1}
I20260926 16:44:13.281692 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:13.245440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:56403 (local address 127.9.238.253:38171)
0926 16:44:13.246019 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:13.246024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:13.247695 (+ 1671us) server_negotiation.cc:407] Connection header received
0926 16:44:13.251599 (+ 3904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:13.251609 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:13.251737 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:13.251841 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:13.255475 (+ 3634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.257265 (+ 1790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.261798 (+ 4533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.262811 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.262952 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:13.273530 (+ 10578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:13.273541 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:13.273548 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:13.273599 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:13.277560 (+ 3961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:13.278666 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:13.278672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:13.278676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:13.278810 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:13.279638 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:13.279643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:13.279646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:13.279981 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:13.280093 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:13.281086 (+ 993us) server_negotiation.cc:299] Negotiation successful
0926 16:44:13.281316 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":289,"threads_started":1}
I20260926 16:44:13.283224 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:13.246879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:38171 (local address 127.9.238.193:56403)
0926 16:44:13.247606 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:13.247625 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:13.251199 (+ 3574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:13.252652 (+ 1453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:13.252674 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:13.254019 (+ 1345us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:13.255142 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:13.255211 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.257603 (+ 2392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.257613 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:13.261517 (+ 3904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:13.261546 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.263599 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.263609 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:13.263873 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:13.264865 (+ 992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:13.264918 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:13.273183 (+ 8265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:13.277731 (+ 4548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:13.277735 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:13.277738 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:13.278530 (+ 792us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:13.279290 (+ 760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:13.279294 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:13.279297 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:13.279475 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:13.280184 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:13.280189 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:13.280776 (+ 587us) client_negotiation.cc:769] Sending connection context
0926 16:44:13.281864 (+ 1088us) client_negotiation.cc:240] Negotiation successful
0926 16:44:13.282389 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"mutex_wait_us":74,"spinlock_wait_cycles":70144,"thread_start_us":440,"threads_started":1}
I20260926 16:44:13.283659 17331 heartbeater.cc:347] Connected to a master server at 127.9.238.252:34171
I20260926 16:44:13.284173 17331 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:13.284175 17333 heartbeater.cc:347] Connected to a master server at 127.9.238.253:38171
I20260926 16:44:13.284385 17333 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:13.285460 17331 heartbeater.cc:511] Master 127.9.238.252:34171 requested a full tablet report, sending...
I20260926 16:44:13.285506 17333 heartbeater.cc:511] Master 127.9.238.253:38171 requested a full tablet report, sending...
I20260926 16:44:13.288332 16955 ts_manager.cc:194] Registered new tserver with Master: e7a44bfa35494003aadce77ec43974a8 (127.9.238.193:40747)
I20260926 16:44:13.288327 17146 ts_manager.cc:194] Registered new tserver with Master: e7a44bfa35494003aadce77ec43974a8 (127.9.238.193:40747)
Sep 26 16:44:13 dist-test-slave-wt2j krb5kdc[16303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.238.194@KRBTEST.COM
I20260926 16:44:13.294343 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:13.248148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.9.238.193:43571)
0926 16:44:13.248852 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:13.248868 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:13.251146 (+ 2278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:13.255048 (+ 3902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:13.255056 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:13.267618 (+ 12562us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:13.268114 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:13.268133 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.270912 (+ 2779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.270922 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:13.273603 (+ 2681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:13.273632 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.276526 (+ 2894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.276533 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:13.276741 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:13.277611 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:13.277638 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:13.283182 (+ 5544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:13.291174 (+ 7992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:13.291178 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:13.291181 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:13.291749 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:13.292326 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:13.292329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:13.292332 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:13.292482 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:13.293320 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:13.293324 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:13.293467 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 16:44:13.293632 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 16:44:13.293928 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":402,"threads_started":1}
I20260926 16:44:13.294654 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:13.248334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:43571 (local address 127.9.238.254:40511)
0926 16:44:13.254227 (+ 5893us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:13.254236 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:13.254292 (+ 56us) server_negotiation.cc:407] Connection header received
0926 16:44:13.254533 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:13.254542 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:13.254741 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:13.258810 (+ 4069us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:13.268441 (+ 9631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.270221 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.273941 (+ 3720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.276214 (+ 2273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.276441 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:13.285451 (+ 9010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:13.285461 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:13.285469 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:13.285533 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:13.290945 (+ 5412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:13.291934 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:13.291942 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:13.291948 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:13.292163 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:13.292650 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:13.292659 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:13.292666 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:13.293149 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:13.293394 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:13.293803 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 16:44:13.294092 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5704,"thread_start_us":5537,"threads_started":1}
I20260926 16:44:13.295835 17332 heartbeater.cc:347] Connected to a master server at 127.9.238.254:40511
I20260926 16:44:13.296080 17332 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:13.296867 17332 heartbeater.cc:511] Master 127.9.238.254:40511 requested a full tablet report, sending...
I20260926 16:44:13.298578 16347 ts_manager.cc:194] Re-registered known tserver with Master: e7a44bfa35494003aadce77ec43974a8 (127.9.238.193:40747)
I20260926 16:44:13.299847 16347 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.238.193@KRBTEST.COM'} at 127.9.238.193:43571
I20260926 16:44:13.304256 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:13.254439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33588 (local address 127.9.238.194:34685)
0926 16:44:13.269659 (+ 15220us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:13.269668 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:13.275456 (+ 5788us) server_negotiation.cc:407] Connection header received
0926 16:44:13.276948 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:13.276964 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:13.277402 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:13.277618 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:13.281538 (+ 3920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.283777 (+ 2239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.286395 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.287803 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.287975 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:13.295825 (+ 7850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:13.295842 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:13.295859 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:13.295898 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:13.300231 (+ 4333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:13.301459 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:13.301469 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:13.301473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:13.301679 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:13.302259 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:13.302265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:13.302267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:13.302720 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:13.303081 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:13.303462 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 16:44:13.303830 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14999,"thread_start_us":15230,"threads_started":1}
I20260926 16:44:13.305788 16666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:13.408119 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 16543
I20260926 16:44:13.426081 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:34685
--local_ip_for_outbound_sockets=127.9.238.194
--tserver_master_addrs=127.9.238.254:40511
--webserver_port=33303
--webserver_interface=127.9.238.194
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.194.keytab
--principal=oryx/127.9.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:39619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.238.254:40511,127.9.238.253:38171
--tserver_master_addrs=127.9.238.254:40511,127.9.238.253:38171,127.9.238.252:34171 with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:13.767128 17343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:13.767438 17343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:13.767561 17343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:13.775018 17343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:13.775151 17343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:13.775336 17343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:44:13.786423 17343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:39619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.194.keytab
--principal=oryx/127.9.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:34685
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=33303
--tserver_master_addrs=127.9.238.254:40511,127.9.238.253:38171,127.9.238.252:34171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:13.787873 17343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:13.789163 17343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:13.804920 17349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:13.804879 17352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:13.805308 17350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:13.806267 17343 server_base.cc:1034] running on GCE node
I20260926 16:44:13.806967 17343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:13.808099 17343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:13.809384 17343 hybrid_clock.cc:648] HybridClock initialized: now 1790441053809302 us; error 72 us; skew 500 ppm
I20260926 16:44:13.809769 17343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:13 dist-test-slave-wt2j krb5kdc[16303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441053, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:13.813690 17343 init.cc:377] Logged in from keytab as oryx/127.9.238.194@KRBTEST.COM (short username oryx)
I20260926 16:44:13.815536 17343 webserver.cc:460] Webserver started at http://127.9.238.194:33303/ using document root <none> and password file <none>
I20260926 16:44:13.816429 17343 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:13.816622 17343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:13.823205 17343 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 16:44:13.826640 17359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:13.828555 17343 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260926 16:44:13.828773 17343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "b9822a16ad8646d78ff6a980fd5dc5b6"
format_stamp: "Formatted at 2026-09-26 16:44:10 on dist-test-slave-wt2j"
server_key: "77ba6c8eff7f6b7b11ba6a895e5aff93"
server_key_iv: "88bb0d771af4dfe15e40acbd4d8c5a57"
server_key_version: "encryptionkey@0"
I20260926 16:44:13.829833 17343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:13.852900 17343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:13.859483 17343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:13.859841 17343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:13.861224 17343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:13.862831 17343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:13.862962 17343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:13.863075 17343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:13.863193 17343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:13.906374 17343 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:34685
I20260926 16:44:13.906431 17471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:34685 every 8 connection(s)
I20260926 16:44:13.907898 17343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:44:13.914659 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 17343
I20260926 16:44:13.914955 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 16677
Sep 26 16:44:13 dist-test-slave-wt2j krb5kdc[16303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441053, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.252@KRBTEST.COM
Sep 26 16:44:13 dist-test-slave-wt2j krb5kdc[16303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441053, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.253@KRBTEST.COM
Sep 26 16:44:13 dist-test-slave-wt2j krb5kdc[16303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441053, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM
I20260926 16:44:13.945142 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:13.911491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:38771 (local address 127.9.238.252:34171)
0926 16:44:13.912198 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:13.912208 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:13.913553 (+ 1345us) server_negotiation.cc:407] Connection header received
0926 16:44:13.916396 (+ 2843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:13.916402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:13.916514 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:13.916627 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:13.919381 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.920817 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.924144 (+ 3327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.924915 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.925055 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:13.931747 (+ 6692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:13.931757 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:13.931764 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:13.931819 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:13.937093 (+ 5274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:13.938511 (+ 1418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:13.938520 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:13.938525 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:13.938722 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:13.940999 (+ 2277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:13.941005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:13.941009 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:13.941592 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:13.941766 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:13.944193 (+ 2427us) server_negotiation.cc:299] Negotiation successful
0926 16:44:13.944553 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":319,"threads_started":1}
I20260926 16:44:13.947463 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:13.914112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:60341 (local address 127.9.238.253:38171)
0926 16:44:13.924013 (+ 9901us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:13.924022 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:13.925016 (+ 994us) server_negotiation.cc:407] Connection header received
0926 16:44:13.925122 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:13.925132 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:13.925300 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:13.925447 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:13.927810 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.929064 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.931112 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.931839 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.931954 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:13.937473 (+ 5519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:13.937482 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:13.937486 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:13.937537 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:13.942370 (+ 4833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:13.943805 (+ 1435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:13.943811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:13.943815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:13.944058 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:13.944569 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:13.944575 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:13.944578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:13.945162 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:13.945747 (+ 585us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:13.946645 (+ 898us) server_negotiation.cc:299] Negotiation successful
0926 16:44:13.946932 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9725,"thread_start_us":9605,"threads_started":1}
I20260926 16:44:13.948580 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:13.912545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:34171 (local address 127.9.238.194:38771)
0926 16:44:13.913372 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:13.913419 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:13.916025 (+ 2606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:13.917275 (+ 1250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:13.917291 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:13.918189 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:13.919136 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:13.919165 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.921074 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.921082 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:13.923388 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:13.923411 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.925493 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.925503 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:13.925922 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:13.927011 (+ 1089us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:13.927059 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:13.931322 (+ 4263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:13.937301 (+ 5979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:13.937314 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:13.937343 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:13.938319 (+ 976us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:13.938920 (+ 601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:13.938924 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:13.938928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:13.940758 (+ 1830us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:13.942663 (+ 1905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:13.942675 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:13.943489 (+ 814us) client_negotiation.cc:769] Sending connection context
0926 16:44:13.945564 (+ 2075us) client_negotiation.cc:240] Negotiation successful
0926 16:44:13.946497 (+ 933us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":1429,"threads_started":1}
I20260926 16:44:13.949258 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:13.912158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.9.238.194:43417)
0926 16:44:13.913383 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:13.913428 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:13.920516 (+ 7088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:13.927662 (+ 7146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:13.927669 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:13.928367 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:13.928796 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:13.928808 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.931278 (+ 2470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.931283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:13.932929 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:13.932944 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.934432 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.934442 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:13.934800 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:13.937443 (+ 2643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:13.937480 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:13.941248 (+ 3768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:13.946012 (+ 4764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:13.946018 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:13.946023 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:13.946778 (+ 755us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:13.947239 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:13.947244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:13.947253 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:13.947427 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:13.948105 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:13.948110 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:13.948310 (+ 200us) client_negotiation.cc:769] Sending connection context
0926 16:44:13.948439 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 16:44:13.948757 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"thread_start_us":501,"threads_started":1}
I20260926 16:44:13.949749 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:13.913854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:38171 (local address 127.9.238.194:60341)
0926 16:44:13.924624 (+ 10770us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:13.924656 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:13.924857 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:13.925750 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:13.925758 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:13.927143 (+ 1385us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:13.927611 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:13.927627 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.929255 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.929259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:13.930853 (+ 1594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:13.930868 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.932354 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.932360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:13.932544 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:13.933851 (+ 1307us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:13.933879 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:13.937165 (+ 3286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:13.942647 (+ 5482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:13.942654 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:13.942658 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:13.943534 (+ 876us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:13.944236 (+ 702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:13.944241 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:13.944245 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:13.944412 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:13.945820 (+ 1408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:13.945826 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:13.946021 (+ 195us) client_negotiation.cc:769] Sending connection context
0926 16:44:13.946263 (+ 242us) client_negotiation.cc:240] Negotiation successful
0926 16:44:13.948397 (+ 2134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10561,"mutex_wait_us":1727,"spinlock_wait_cycles":28544,"thread_start_us":10316,"threads_started":1}
I20260926 16:44:13.950892 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:13.911486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:43417 (local address 127.9.238.254:40511)
0926 16:44:13.927044 (+ 15558us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:13.927051 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:13.927074 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:44:13.927144 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:13.927149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:13.927241 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:13.929017 (+ 1776us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:13.930079 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.931123 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.933174 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.934177 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.934335 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:13.941614 (+ 7279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:13.941627 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:13.941633 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:13.941716 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:13.945838 (+ 4122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:13.946929 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:13.946935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:13.946939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:13.947093 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:13.947598 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:13.947603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:13.947607 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:13.947952 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:13.950118 (+ 2166us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:13.950251 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 16:44:13.950490 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15396,"thread_start_us":15902,"threads_started":1}
I20260926 16:44:13.950969 17472 heartbeater.cc:347] Connected to a master server at 127.9.238.252:34171
I20260926 16:44:13.951447 17472 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:13.951807 17474 heartbeater.cc:347] Connected to a master server at 127.9.238.253:38171
I20260926 16:44:13.952008 17474 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:13.952814 17473 heartbeater.cc:347] Connected to a master server at 127.9.238.254:40511
I20260926 16:44:13.952943 17473 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:13.953198 17472 heartbeater.cc:511] Master 127.9.238.252:34171 requested a full tablet report, sending...
I20260926 16:44:13.953212 17474 heartbeater.cc:511] Master 127.9.238.253:38171 requested a full tablet report, sending...
I20260926 16:44:13.953502 17473 heartbeater.cc:511] Master 127.9.238.254:40511 requested a full tablet report, sending...
I20260926 16:44:13.955344 16347 ts_manager.cc:194] Re-registered known tserver with Master: b9822a16ad8646d78ff6a980fd5dc5b6 (127.9.238.194:34685)
I20260926 16:44:13.955806 17146 ts_manager.cc:194] Registered new tserver with Master: b9822a16ad8646d78ff6a980fd5dc5b6 (127.9.238.194:34685)
I20260926 16:44:13.955837 16955 ts_manager.cc:194] Registered new tserver with Master: b9822a16ad8646d78ff6a980fd5dc5b6 (127.9.238.194:34685)
Sep 26 16:44:13 dist-test-slave-wt2j krb5kdc[16303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.238.195@KRBTEST.COM
I20260926 16:44:13.956571 16347 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.238.194@KRBTEST.COM'} at 127.9.238.194:43417
I20260926 16:44:13.963814 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:13.916296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37382 (local address 127.9.238.195:42613)
0926 16:44:13.940146 (+ 23850us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:13.940160 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:13.941806 (+ 1646us) server_negotiation.cc:407] Connection header received
0926 16:44:13.941885 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:13.941895 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:13.942137 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:13.942327 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:13.944589 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.946261 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.948121 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:13.948997 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:13.949121 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:13.957197 (+ 8076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:13.957215 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:13.957235 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:13.957294 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:13.960348 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:13.961430 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:13.961437 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:13.961440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:13.961587 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:13.961963 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:13.961966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:13.961968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:13.962539 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:13.962771 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:13.963175 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 16:44:13.963380 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23623,"thread_start_us":23453,"threads_started":1}
I20260926 16:44:13.965442 16800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:14.068459 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 16677
W20260926 16:44:14.087148 16334 connection.cc:535] server connection from 127.9.238.195:39777 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 16:44:14.088264 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:42613
--local_ip_for_outbound_sockets=127.9.238.195
--tserver_master_addrs=127.9.238.254:40511
--webserver_port=33733
--webserver_interface=127.9.238.195
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.195.keytab
--principal=oryx/127.9.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:39619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.9.238.254:40511,127.9.238.253:38171
--tserver_master_addrs=127.9.238.254:40511,127.9.238.253:38171,127.9.238.252:34171 with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:44:14.303037 17332 heartbeater.cc:503] Master 127.9.238.254:40511 was elected leader, sending a full tablet report...
W20260926 16:44:14.435643 17485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:14.436029 17485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:14.436172 17485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:14.445055 17485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:14.445257 17485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:14.445451 17485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:44:14.457394 17485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:39619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.195.keytab
--principal=oryx/127.9.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:42613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=33733
--tserver_master_addrs=127.9.238.254:40511,127.9.238.253:38171,127.9.238.252:34171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:14.458899 17485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:14.460404 17485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:14.476197 17490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:14.476263 17491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:14.476207 17493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:14.477691 17485 server_base.cc:1034] running on GCE node
I20260926 16:44:14.478420 17485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:14.480000 17485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:14.481312 17485 hybrid_clock.cc:648] HybridClock initialized: now 1790441054481263 us; error 60 us; skew 500 ppm
I20260926 16:44:14.481812 17485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:14 dist-test-slave-wt2j krb5kdc[16303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441054, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:14.498032 17485 init.cc:377] Logged in from keytab as oryx/127.9.238.195@KRBTEST.COM (short username oryx)
I20260926 16:44:14.500159 17485 webserver.cc:460] Webserver started at http://127.9.238.195:33733/ using document root <none> and password file <none>
I20260926 16:44:14.500968 17485 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:14.501130 17485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:14.506886 17485 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 16:44:14.510113 17500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:14.512071 17485 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:44:14.512261 17485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "ee01f03feda544c59ff2dd4f5467404b"
format_stamp: "Formatted at 2026-09-26 16:44:10 on dist-test-slave-wt2j"
server_key: "bb63c891024b81999442df12511dd200"
server_key_iv: "f45fa4866c9e88a17979020fdbc8696b"
server_key_version: "encryptionkey@0"
I20260926 16:44:14.513406 17485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:14.531229 17485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:14.537194 17485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:14.537565 17485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:14.538769 17485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:14.540632 17485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:14.540755 17485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:14.540904 17485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:14.541005 17485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:14.584553 17485 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:42613
I20260926 16:44:14.584604 17612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:42613 every 8 connection(s)
I20260926 16:44:14.586117 17485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:44:14.593092 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 17485
Sep 26 16:44:14 dist-test-slave-wt2j krb5kdc[16303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441054, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.252@KRBTEST.COM
Sep 26 16:44:14 dist-test-slave-wt2j krb5kdc[16303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441054, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.253@KRBTEST.COM
Sep 26 16:44:14 dist-test-slave-wt2j krb5kdc[16303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441054, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM
I20260926 16:44:14.625551 17618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:14.590008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:35397 (local address 127.9.238.252:34171)
0926 16:44:14.591054 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:14.591060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:14.592267 (+ 1207us) server_negotiation.cc:407] Connection header received
0926 16:44:14.599320 (+ 7053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:14.599329 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:14.599423 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:14.599512 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:14.602228 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.603320 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.605906 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.606592 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.606696 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:14.613832 (+ 7136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:14.613841 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:14.613846 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:14.613918 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:14.618154 (+ 4236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:14.619434 (+ 1280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:14.619445 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:14.619451 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:14.619713 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:14.620496 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:14.620501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:14.620504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:14.620965 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:14.621145 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:14.624826 (+ 3681us) server_negotiation.cc:299] Negotiation successful
0926 16:44:14.625126 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":934,"thread_start_us":820,"threads_started":1}
I20260926 16:44:14.626144 17617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:14.590004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:33259 (local address 127.9.238.253:38171)
0926 16:44:14.590680 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:14.590690 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:14.592267 (+ 1577us) server_negotiation.cc:407] Connection header received
0926 16:44:14.597104 (+ 4837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:14.597116 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:14.597266 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:14.597399 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:14.601453 (+ 4054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.602779 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.605901 (+ 3122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.606574 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.606691 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:14.614727 (+ 8036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:14.614737 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:14.614746 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:14.614829 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:14.619901 (+ 5072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:14.621787 (+ 1886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:14.621797 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:14.621803 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:14.622032 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:14.624093 (+ 2061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:14.624098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:14.624102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:14.624593 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:14.624818 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:14.625221 (+ 403us) server_negotiation.cc:299] Negotiation successful
0926 16:44:14.625572 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":290,"threads_started":1}
I20260926 16:44:14.626873 17619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:14.590963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:38171 (local address 127.9.238.195:33259)
0926 16:44:14.592079 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:14.592125 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:14.596556 (+ 4431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:14.598237 (+ 1681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:14.598257 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:14.599929 (+ 1672us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:14.601153 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:14.601188 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.603007 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.603012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:14.605708 (+ 2696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:14.605724 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.608501 (+ 2777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.608511 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:14.608925 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:14.611338 (+ 2413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:14.611366 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:14.614383 (+ 3017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:14.620072 (+ 5689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:14.620076 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:14.620080 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:14.621623 (+ 1543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:14.623578 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:14.623583 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:14.623587 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:14.623961 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:14.624728 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:14.624733 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:14.624912 (+ 179us) client_negotiation.cc:769] Sending connection context
0926 16:44:14.625056 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 16:44:14.625381 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"mutex_wait_us":92,"thread_start_us":463,"threads_started":1}
I20260926 16:44:14.627027 17620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:14.590630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:34171 (local address 127.9.238.195:35397)
0926 16:44:14.592064 (+ 1434us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:14.592126 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:14.598956 (+ 6830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:14.599806 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:14.599812 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:14.600455 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:14.601737 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:14.601747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.603653 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.603658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:14.605711 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:14.605724 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.607710 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.607716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:14.607978 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:14.609360 (+ 1382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:14.609396 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:14.613510 (+ 4114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:14.618364 (+ 4854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:14.618377 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:14.618409 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:14.619217 (+ 808us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:14.619874 (+ 657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:14.619883 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:14.619890 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:14.620205 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:14.621275 (+ 1070us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:14.621287 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:14.621916 (+ 629us) client_negotiation.cc:769] Sending connection context
0926 16:44:14.622323 (+ 407us) client_negotiation.cc:240] Negotiation successful
0926 16:44:14.623026 (+ 703us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"mutex_wait_us":1016,"thread_start_us":1043,"threads_started":1}
I20260926 16:44:14.629172 17613 heartbeater.cc:347] Connected to a master server at 127.9.238.252:34171
I20260926 16:44:14.629578 17613 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:14.630722 17613 heartbeater.cc:511] Master 127.9.238.252:34171 requested a full tablet report, sending...
I20260926 16:44:14.633440 17146 ts_manager.cc:194] Registered new tserver with Master: ee01f03feda544c59ff2dd4f5467404b (127.9.238.195:42613)
I20260926 16:44:14.637859 17622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:14.592913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:55789 (local address 127.9.238.254:40511)
0926 16:44:14.602735 (+ 9822us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:14.602745 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:14.609591 (+ 6846us) server_negotiation.cc:407] Connection header received
0926 16:44:14.609811 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:14.609820 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:14.609953 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:14.610154 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:14.612434 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.613950 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.616684 (+ 2734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.617521 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.617674 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:14.629520 (+ 11846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:14.629528 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:14.629532 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:14.629585 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:14.634603 (+ 5018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:14.635480 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:14.635491 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:14.635498 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:14.635726 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:14.636137 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:14.636142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:14.636147 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:14.636632 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:14.636782 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:14.637194 (+ 412us) server_negotiation.cc:299] Negotiation successful
0926 16:44:14.637451 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9623,"thread_start_us":10499,"threads_started":1}
I20260926 16:44:14.637908 17621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:14.592771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.9.238.195:55789)
0926 16:44:14.609462 (+ 16691us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:14.609488 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:14.609646 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:14.610187 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:14.610193 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:14.610782 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:14.611265 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:14.611287 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.614633 (+ 3346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.614639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:14.616392 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:14.616408 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.618012 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.618022 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:14.618230 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:14.621972 (+ 3742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:14.621997 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:14.629247 (+ 7250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:14.634778 (+ 5531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:14.634784 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:14.634788 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:14.635321 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:14.635834 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:14.635840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:14.635854 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:14.636016 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:14.636749 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:14.636754 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:14.636939 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 16:44:14.637085 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 16:44:14.637433 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16249,"thread_start_us":15923,"threads_started":1}
I20260926 16:44:14.639070 17615 heartbeater.cc:347] Connected to a master server at 127.9.238.253:38171
I20260926 16:44:14.639277 17615 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:14.639871 17615 heartbeater.cc:511] Master 127.9.238.253:38171 requested a full tablet report, sending...
I20260926 16:44:14.640547 17614 heartbeater.cc:347] Connected to a master server at 127.9.238.254:40511
I20260926 16:44:14.640662 17614 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:14.641129 17614 heartbeater.cc:511] Master 127.9.238.254:40511 requested a full tablet report, sending...
I20260926 16:44:14.642220 16347 ts_manager.cc:194] Re-registered known tserver with Master: ee01f03feda544c59ff2dd4f5467404b (127.9.238.195:42613)
I20260926 16:44:14.642452 16955 ts_manager.cc:194] Registered new tserver with Master: ee01f03feda544c59ff2dd4f5467404b (127.9.238.195:42613)
I20260926 16:44:14.643522 16347 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.238.195@KRBTEST.COM'} at 127.9.238.195:55789
I20260926 16:44:14.657047 17617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:14.624031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52138 (local address 127.9.238.253:38171)
0926 16:44:14.626844 (+ 2813us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:14.626850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:14.635647 (+ 8797us) server_negotiation.cc:407] Connection header received
0926 16:44:14.635824 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:14.635834 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:14.636009 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:14.636140 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:14.639708 (+ 3568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.640760 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.643833 (+ 3073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.644435 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.644548 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:14.649409 (+ 4861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:14.649415 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:14.649419 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:14.649453 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:14.653478 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:14.654329 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:14.654335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:14.654339 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:14.654552 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:14.655070 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:14.655076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:14.655079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:14.655450 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:14.655754 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:14.656417 (+ 663us) server_negotiation.cc:299] Negotiation successful
0926 16:44:14.656660 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2680,"thread_start_us":14122,"threads_started":1}
I20260926 16:44:14.657428 17629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:14.620524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39028 (local address 127.9.238.254:40511)
0926 16:44:14.621491 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:14.621501 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:14.628818 (+ 7317us) server_negotiation.cc:407] Connection header received
0926 16:44:14.628908 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:14.628917 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:14.629031 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:14.629154 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:14.638462 (+ 9308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.640040 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.644019 (+ 3979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.644588 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.644763 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:14.649379 (+ 4616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:14.649388 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:14.649393 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:14.649442 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:14.653577 (+ 4135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:14.654736 (+ 1159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:14.654742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:14.654746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:14.654904 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:14.655440 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:14.655448 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:14.655452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:14.655895 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:14.656190 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:14.656553 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 16:44:14.656808 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":794,"thread_start_us":616,"threads_started":1}
I20260926 16:44:14.659224 17618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:14.618215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60448 (local address 127.9.238.252:34171)
0926 16:44:14.626203 (+ 7988us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:14.626209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:14.626226 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:44:14.626280 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:14.626285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:14.626379 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:14.626482 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:14.643692 (+ 17210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.645187 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.647506 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.648332 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.648514 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:14.651505 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:14.651510 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:14.651514 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:14.651607 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:14.654897 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:14.655782 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:14.655788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:14.655793 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:14.656000 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:14.656482 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:14.656488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:14.656491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:14.656862 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:14.657095 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:14.658664 (+ 1569us) server_negotiation.cc:299] Negotiation successful
0926 16:44:14.658851 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7872,"thread_start_us":8583,"threads_started":1}
I20260926 16:44:14.662708 16347 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:44:14.670586 16347 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:44:14.721359 17646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:14.692677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:34685 (local address 127.0.0.1:33592)
0926 16:44:14.707719 (+ 15042us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:14.707741 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:14.707921 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:14.713035 (+ 5114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:14.713040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:14.713066 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:14.713469 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:14.713483 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.715690 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.715695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:14.718392 (+ 2697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:14.718410 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.720497 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.720504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:14.720684 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:14.720733 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 16:44:14.720875 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 16:44:14.721038 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14906,"thread_start_us":14717,"threads_started":1}
I20260926 16:44:14.722090 17647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:14.692882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:42613 (local address 127.0.0.1:37392)
0926 16:44:14.712764 (+ 19882us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:14.712785 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:14.712966 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:14.713843 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:14.713848 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:14.713872 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:14.714318 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:14.714334 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.716735 (+ 2401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.716739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:14.719055 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:14.719071 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.721113 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.721119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:14.721319 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:14.721362 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 16:44:14.721468 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 16:44:14.721622 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19726,"mutex_wait_us":66,"thread_start_us":19563,"threads_started":1}
I20260926 16:44:14.722203 17648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:14.692957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33592 (local address 127.9.238.194:34685)
0926 16:44:14.711857 (+ 18900us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:14.711868 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:14.711907 (+ 39us) server_negotiation.cc:407] Connection header received
0926 16:44:14.712044 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:14.712054 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:14.712484 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:14.712764 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:14.713702 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.715311 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.718612 (+ 3301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.720255 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.720422 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:14.720690 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:14.721728 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0926 16:44:14.721923 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18677,"thread_start_us":18491,"threads_started":1}
I20260926 16:44:14.722904 17644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:14.691771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:40747 (local address 127.0.0.1:44266)
0926 16:44:14.692406 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:14.692431 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:14.692588 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:14.714281 (+ 21693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:14.714296 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:14.714335 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:14.714786 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:14.714801 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.716690 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.716696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:14.719921 (+ 3225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:14.719937 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.722201 (+ 2264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.722210 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:14.722437 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:14.722464 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 16:44:14.722514 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 16:44:14.722630 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":542,"thread_start_us":737,"threads_started":1}
I20260926 16:44:14.723853 17645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:14.692358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44266 (local address 127.9.238.193:40747)
0926 16:44:14.713561 (+ 21203us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:14.713572 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:14.713600 (+ 28us) server_negotiation.cc:407] Connection header received
0926 16:44:14.713706 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:14.713711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:14.713964 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:14.714247 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:14.714993 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.716532 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.720097 (+ 3565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.721959 (+ 1862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.723172 (+ 1213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:14.723327 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:14.723394 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 16:44:14.723596 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21002,"thread_start_us":20855,"threads_started":1}
I20260926 16:44:14.724066 17649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:14.693326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37392 (local address 127.9.238.195:42613)
0926 16:44:14.706702 (+ 13376us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:14.706721 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:14.712968 (+ 6247us) server_negotiation.cc:407] Connection header received
0926 16:44:14.713212 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:14.713224 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:14.713557 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:14.713860 (+ 303us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:14.714565 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.716521 (+ 1956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.719203 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.720558 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.720698 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:14.720956 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:14.722119 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0926 16:44:14.723681 (+ 1562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13041,"thread_start_us":13414,"threads_started":1}
I20260926 16:44:14.726534 17407 tablet_service.cc:1467] Processing CreateTablet for tablet 951a8e6683de43219d8d400147ae1a01 (DEFAULT_TABLE table=test-table [id=a7f5394bfe9e4431b6c25b7b1170b3bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:14.728453 17407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 951a8e6683de43219d8d400147ae1a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:14.728317 17548 tablet_service.cc:1467] Processing CreateTablet for tablet 951a8e6683de43219d8d400147ae1a01 (DEFAULT_TABLE table=test-table [id=a7f5394bfe9e4431b6c25b7b1170b3bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:14.728317 17266 tablet_service.cc:1467] Processing CreateTablet for tablet 951a8e6683de43219d8d400147ae1a01 (DEFAULT_TABLE table=test-table [id=a7f5394bfe9e4431b6c25b7b1170b3bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:14.729868 17548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 951a8e6683de43219d8d400147ae1a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:14.729910 17266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 951a8e6683de43219d8d400147ae1a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:14.743196 17650 tablet_bootstrap.cc:492] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6: Bootstrap starting.
I20260926 16:44:14.746979 17650 tablet_bootstrap.cc:654] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:14.747407 17652 tablet_bootstrap.cc:492] T 951a8e6683de43219d8d400147ae1a01 P ee01f03feda544c59ff2dd4f5467404b: Bootstrap starting.
I20260926 16:44:14.750214 17650 log.cc:824] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:14.751387 17652 tablet_bootstrap.cc:654] T 951a8e6683de43219d8d400147ae1a01 P ee01f03feda544c59ff2dd4f5467404b: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:14.752560 17652 log.cc:824] T 951a8e6683de43219d8d400147ae1a01 P ee01f03feda544c59ff2dd4f5467404b: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:14.753034 17651 tablet_bootstrap.cc:492] T 951a8e6683de43219d8d400147ae1a01 P e7a44bfa35494003aadce77ec43974a8: Bootstrap starting.
I20260926 16:44:14.755002 17650 tablet_bootstrap.cc:492] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6: No bootstrap required, opened a new log
I20260926 16:44:14.755406 17650 ts_tablet_manager.cc:1397] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260926 16:44:14.757035 17651 tablet_bootstrap.cc:654] T 951a8e6683de43219d8d400147ae1a01 P e7a44bfa35494003aadce77ec43974a8: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:14.758155 17651 log.cc:824] T 951a8e6683de43219d8d400147ae1a01 P e7a44bfa35494003aadce77ec43974a8: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:14.761946 17652 tablet_bootstrap.cc:492] T 951a8e6683de43219d8d400147ae1a01 P ee01f03feda544c59ff2dd4f5467404b: No bootstrap required, opened a new log
I20260926 16:44:14.762327 17652 ts_tablet_manager.cc:1397] T 951a8e6683de43219d8d400147ae1a01 P ee01f03feda544c59ff2dd4f5467404b: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260926 16:44:14.762373 17651 tablet_bootstrap.cc:492] T 951a8e6683de43219d8d400147ae1a01 P e7a44bfa35494003aadce77ec43974a8: No bootstrap required, opened a new log
I20260926 16:44:14.762705 17651 ts_tablet_manager.cc:1397] T 951a8e6683de43219d8d400147ae1a01 P e7a44bfa35494003aadce77ec43974a8: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20260926 16:44:14.768978 17650 raft_consensus.cc:348] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a44bfa35494003aadce77ec43974a8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 40747 } } peers { permanent_uuid: "ee01f03feda544c59ff2dd4f5467404b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 42613 } } peers { permanent_uuid: "b9822a16ad8646d78ff6a980fd5dc5b6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34685 } }
I20260926 16:44:14.769460 17650 raft_consensus.cc:374] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:14.769564 17650 raft_consensus.cc:729] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9822a16ad8646d78ff6a980fd5dc5b6, State: Initialized, Role: FOLLOWER
I20260926 16:44:14.770283 17650 consensus_queue.cc:260] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a44bfa35494003aadce77ec43974a8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 40747 } } peers { permanent_uuid: "ee01f03feda544c59ff2dd4f5467404b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 42613 } } peers { permanent_uuid: "b9822a16ad8646d78ff6a980fd5dc5b6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34685 } }
I20260926 16:44:14.770915 17652 raft_consensus.cc:348] T 951a8e6683de43219d8d400147ae1a01 P ee01f03feda544c59ff2dd4f5467404b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a44bfa35494003aadce77ec43974a8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 40747 } } peers { permanent_uuid: "ee01f03feda544c59ff2dd4f5467404b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 42613 } } peers { permanent_uuid: "b9822a16ad8646d78ff6a980fd5dc5b6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34685 } }
I20260926 16:44:14.771404 17652 raft_consensus.cc:374] T 951a8e6683de43219d8d400147ae1a01 P ee01f03feda544c59ff2dd4f5467404b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:14.771517 17652 raft_consensus.cc:729] T 951a8e6683de43219d8d400147ae1a01 P ee01f03feda544c59ff2dd4f5467404b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee01f03feda544c59ff2dd4f5467404b, State: Initialized, Role: FOLLOWER
I20260926 16:44:14.772317 17650 ts_tablet_manager.cc:1428] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6: Time spent starting tablet: real 0.017s user 0.011s sys 0.005s
I20260926 16:44:14.772380 17652 consensus_queue.cc:260] T 951a8e6683de43219d8d400147ae1a01 P ee01f03feda544c59ff2dd4f5467404b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a44bfa35494003aadce77ec43974a8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 40747 } } peers { permanent_uuid: "ee01f03feda544c59ff2dd4f5467404b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 42613 } } peers { permanent_uuid: "b9822a16ad8646d78ff6a980fd5dc5b6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34685 } }
I20260926 16:44:14.772655 17473 heartbeater.cc:503] Master 127.9.238.254:40511 was elected leader, sending a full tablet report...
I20260926 16:44:14.773598 17651 raft_consensus.cc:348] T 951a8e6683de43219d8d400147ae1a01 P e7a44bfa35494003aadce77ec43974a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a44bfa35494003aadce77ec43974a8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 40747 } } peers { permanent_uuid: "ee01f03feda544c59ff2dd4f5467404b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 42613 } } peers { permanent_uuid: "b9822a16ad8646d78ff6a980fd5dc5b6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34685 } }
I20260926 16:44:14.774036 17651 raft_consensus.cc:374] T 951a8e6683de43219d8d400147ae1a01 P e7a44bfa35494003aadce77ec43974a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:14.774143 17651 raft_consensus.cc:729] T 951a8e6683de43219d8d400147ae1a01 P e7a44bfa35494003aadce77ec43974a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a44bfa35494003aadce77ec43974a8, State: Initialized, Role: FOLLOWER
I20260926 16:44:14.774758 17651 consensus_queue.cc:260] T 951a8e6683de43219d8d400147ae1a01 P e7a44bfa35494003aadce77ec43974a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a44bfa35494003aadce77ec43974a8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 40747 } } peers { permanent_uuid: "ee01f03feda544c59ff2dd4f5467404b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 42613 } } peers { permanent_uuid: "b9822a16ad8646d78ff6a980fd5dc5b6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34685 } }
I20260926 16:44:14.780196 17652 ts_tablet_manager.cc:1428] T 951a8e6683de43219d8d400147ae1a01 P ee01f03feda544c59ff2dd4f5467404b: Time spent starting tablet: real 0.018s user 0.015s sys 0.005s
I20260926 16:44:14.780493 17614 heartbeater.cc:503] Master 127.9.238.254:40511 was elected leader, sending a full tablet report...
I20260926 16:44:14.787938 17651 ts_tablet_manager.cc:1428] T 951a8e6683de43219d8d400147ae1a01 P e7a44bfa35494003aadce77ec43974a8: Time spent starting tablet: real 0.025s user 0.021s sys 0.002s
I20260926 16:44:14.815824 17656 raft_consensus.cc:482] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:44:14.816279 17656 raft_consensus.cc:504] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a44bfa35494003aadce77ec43974a8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 40747 } } peers { permanent_uuid: "ee01f03feda544c59ff2dd4f5467404b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 42613 } } peers { permanent_uuid: "b9822a16ad8646d78ff6a980fd5dc5b6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34685 } }
I20260926 16:44:14.818112 17656 leader_election.cc:288] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7a44bfa35494003aadce77ec43974a8 (127.9.238.193:40747), ee01f03feda544c59ff2dd4f5467404b (127.9.238.195:42613)
I20260926 16:44:14.826179 17659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:14.818559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:40747 (local address 127.9.238.194:60801)
0926 16:44:14.819301 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:14.819318 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:14.819535 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:14.820108 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:14.820112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:14.820137 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:14.820604 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:14.820619 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.822187 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.822193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:14.823937 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:14.823949 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.825319 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.825325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:14.825516 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:14.825548 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 16:44:14.825657 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 16:44:14.825791 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":1,"thread_start_us":845,"threads_started":1}
I20260926 16:44:14.826197 17645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:14.818694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:60801 (local address 127.9.238.193:40747)
0926 16:44:14.818958 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:14.818967 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:14.819443 (+ 476us) server_negotiation.cc:407] Connection header received
0926 16:44:14.819726 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:14.819731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:14.819825 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:14.819915 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:14.820832 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.822025 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.824098 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.825091 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.825277 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:14.825356 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:14.825717 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 16:44:14.825900 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:44:14.827343 17286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951a8e6683de43219d8d400147ae1a01" candidate_uuid: "b9822a16ad8646d78ff6a980fd5dc5b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a44bfa35494003aadce77ec43974a8" is_pre_election: true
I20260926 16:44:14.827689 17649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:14.820027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:56413 (local address 127.9.238.195:42613)
0926 16:44:14.820223 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:14.820230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:14.820790 (+ 560us) server_negotiation.cc:407] Connection header received
0926 16:44:14.821036 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:14.821043 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:14.821103 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:14.821207 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:14.821657 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.822618 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.825520 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.826695 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.826884 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:14.826954 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:14.827234 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 16:44:14.827354 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:44:14.827905 17660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:14.819904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:42613 (local address 127.9.238.194:56413)
0926 16:44:14.820703 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:14.820722 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:14.820879 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:14.821214 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:14.821218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:14.821239 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:14.821514 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:14.821522 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.822791 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.822797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:14.825255 (+ 2458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:14.825277 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.826900 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.826907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:14.827080 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:14.827103 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 16:44:14.827210 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 16:44:14.827354 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":51,"thread_start_us":521,"threads_started":1}
I20260926 16:44:14.828042 17286 raft_consensus.cc:2454] T 951a8e6683de43219d8d400147ae1a01 P e7a44bfa35494003aadce77ec43974a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9822a16ad8646d78ff6a980fd5dc5b6 in term 0.
I20260926 16:44:14.828864 17568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951a8e6683de43219d8d400147ae1a01" candidate_uuid: "b9822a16ad8646d78ff6a980fd5dc5b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee01f03feda544c59ff2dd4f5467404b" is_pre_election: true
I20260926 16:44:14.829021 17362 leader_election.cc:302] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9822a16ad8646d78ff6a980fd5dc5b6, e7a44bfa35494003aadce77ec43974a8; no voters:
I20260926 16:44:14.829428 17568 raft_consensus.cc:2454] T 951a8e6683de43219d8d400147ae1a01 P ee01f03feda544c59ff2dd4f5467404b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9822a16ad8646d78ff6a980fd5dc5b6 in term 0.
I20260926 16:44:14.829545 17656 raft_consensus.cc:2781] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:44:14.829722 17656 raft_consensus.cc:482] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:44:14.829816 17656 raft_consensus.cc:3037] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:14.832823 17656 raft_consensus.cc:504] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a44bfa35494003aadce77ec43974a8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 40747 } } peers { permanent_uuid: "ee01f03feda544c59ff2dd4f5467404b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 42613 } } peers { permanent_uuid: "b9822a16ad8646d78ff6a980fd5dc5b6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34685 } }
I20260926 16:44:14.833983 17656 leader_election.cc:288] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6 [CANDIDATE]: Term 1 election: Requested vote from peers e7a44bfa35494003aadce77ec43974a8 (127.9.238.193:40747), ee01f03feda544c59ff2dd4f5467404b (127.9.238.195:42613)
I20260926 16:44:14.834615 17568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951a8e6683de43219d8d400147ae1a01" candidate_uuid: "b9822a16ad8646d78ff6a980fd5dc5b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee01f03feda544c59ff2dd4f5467404b"
I20260926 16:44:14.834643 17286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "951a8e6683de43219d8d400147ae1a01" candidate_uuid: "b9822a16ad8646d78ff6a980fd5dc5b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a44bfa35494003aadce77ec43974a8"
I20260926 16:44:14.834906 17568 raft_consensus.cc:3037] T 951a8e6683de43219d8d400147ae1a01 P ee01f03feda544c59ff2dd4f5467404b [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:14.834892 17286 raft_consensus.cc:3037] T 951a8e6683de43219d8d400147ae1a01 P e7a44bfa35494003aadce77ec43974a8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:14.838586 17286 raft_consensus.cc:2454] T 951a8e6683de43219d8d400147ae1a01 P e7a44bfa35494003aadce77ec43974a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9822a16ad8646d78ff6a980fd5dc5b6 in term 1.
I20260926 16:44:14.838585 17568 raft_consensus.cc:2454] T 951a8e6683de43219d8d400147ae1a01 P ee01f03feda544c59ff2dd4f5467404b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9822a16ad8646d78ff6a980fd5dc5b6 in term 1.
I20260926 16:44:14.839107 17362 leader_election.cc:302] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9822a16ad8646d78ff6a980fd5dc5b6, e7a44bfa35494003aadce77ec43974a8; no voters:
I20260926 16:44:14.839434 17656 raft_consensus.cc:2781] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:44:14.840183 17656 raft_consensus.cc:686] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6 [term 1 LEADER]: Becoming Leader. State: Replica: b9822a16ad8646d78ff6a980fd5dc5b6, State: Running, Role: LEADER
W20260926 16:44:14.840210 17616 tablet.cc:2369] T 951a8e6683de43219d8d400147ae1a01 P ee01f03feda544c59ff2dd4f5467404b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:44:14.840870 17656 consensus_queue.cc:237] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a44bfa35494003aadce77ec43974a8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 40747 } } peers { permanent_uuid: "ee01f03feda544c59ff2dd4f5467404b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 42613 } } peers { permanent_uuid: "b9822a16ad8646d78ff6a980fd5dc5b6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34685 } }
I20260926 16:44:14.847085 16347 catalog_manager.cc:5495] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b9822a16ad8646d78ff6a980fd5dc5b6 (127.9.238.194). New cstate: current_term: 1 leader_uuid: "b9822a16ad8646d78ff6a980fd5dc5b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7a44bfa35494003aadce77ec43974a8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee01f03feda544c59ff2dd4f5467404b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 42613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9822a16ad8646d78ff6a980fd5dc5b6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34685 } health_report { overall_health: HEALTHY } } }
I20260926 16:44:14.878553 17648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:14.871702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33606 (local address 127.9.238.194:34685)
0926 16:44:14.871943 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:14.871948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:14.872254 (+ 306us) server_negotiation.cc:407] Connection header received
0926 16:44:14.872499 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:14.872505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:14.872611 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:14.872775 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:44:14.873236 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.874074 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.875872 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.876385 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.876508 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:14.877137 (+ 629us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:44:14.877871 (+ 734us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:44:14.878006 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:14.878223 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 16:44:14.878296 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 16:44:14.889186 17568 raft_consensus.cc:1261] T 951a8e6683de43219d8d400147ae1a01 P ee01f03feda544c59ff2dd4f5467404b [term 1 FOLLOWER]: Refusing update from remote peer b9822a16ad8646d78ff6a980fd5dc5b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:44:14.889186 17286 raft_consensus.cc:1261] T 951a8e6683de43219d8d400147ae1a01 P e7a44bfa35494003aadce77ec43974a8 [term 1 FOLLOWER]: Refusing update from remote peer b9822a16ad8646d78ff6a980fd5dc5b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:44:14.890615 17661 consensus_queue.cc:1035] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee01f03feda544c59ff2dd4f5467404b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 42613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:44:14.891098 17656 consensus_queue.cc:1035] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7a44bfa35494003aadce77ec43974a8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:44:14.910177 17664 mvcc.cc:205] Tried to move back new op lower bound from 7333646560811241472 to 7333646560633241600. Current Snapshot: MvccSnapshot[applied={T|T < 7333646560811241472}]
I20260926 16:44:14.915354 17665 mvcc.cc:205] Tried to move back new op lower bound from 7333646560811241472 to 7333646560633241600. Current Snapshot: MvccSnapshot[applied={T|T < 7333646560811241472}]
I20260926 16:44:14.949512 16346 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:44:14.949836 16346 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:44:14.958920 16346 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 212c6cc5c6e6456b919eb5680dc91f11: Sending DeleteTablet for 3 replicas of tablet 951a8e6683de43219d8d400147ae1a01
I20260926 16:44:14.960807 17548 tablet_service.cc:1514] Processing DeleteTablet for tablet 951a8e6683de43219d8d400147ae1a01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:14 UTC) from {username='oryx', principal='oryx/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:37392
I20260926 16:44:14.961005 17266 tablet_service.cc:1514] Processing DeleteTablet for tablet 951a8e6683de43219d8d400147ae1a01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:14 UTC) from {username='oryx', principal='oryx/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:44266
I20260926 16:44:14.961843 17407 tablet_service.cc:1514] Processing DeleteTablet for tablet 951a8e6683de43219d8d400147ae1a01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:14 UTC) from {username='oryx', principal='oryx/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:33592
I20260926 16:44:14.963039 17676 tablet_replica.cc:330] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6: stopping tablet replica
I20260926 16:44:14.963688 17676 raft_consensus.cc:2229] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:44:14.964512 17676 raft_consensus.cc:2258] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:44:14.967267 17676 ts_tablet_manager.cc:1905] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:44:14.976154 17674 tablet_replica.cc:330] T 951a8e6683de43219d8d400147ae1a01 P ee01f03feda544c59ff2dd4f5467404b: stopping tablet replica
I20260926 16:44:14.976812 17674 raft_consensus.cc:2229] T 951a8e6683de43219d8d400147ae1a01 P ee01f03feda544c59ff2dd4f5467404b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:44:14.977494 17674 raft_consensus.cc:2258] T 951a8e6683de43219d8d400147ae1a01 P ee01f03feda544c59ff2dd4f5467404b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:44:14.978399 17676 ts_tablet_manager.cc:1918] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:44:14.978564 17675 tablet_replica.cc:330] T 951a8e6683de43219d8d400147ae1a01 P e7a44bfa35494003aadce77ec43974a8: stopping tablet replica
I20260926 16:44:14.978639 17676 log.cc:1196] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/wal/wals/951a8e6683de43219d8d400147ae1a01
I20260926 16:44:14.979071 17675 raft_consensus.cc:2229] T 951a8e6683de43219d8d400147ae1a01 P e7a44bfa35494003aadce77ec43974a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:44:14.979470 17676 ts_tablet_manager.cc:1939] T 951a8e6683de43219d8d400147ae1a01 P b9822a16ad8646d78ff6a980fd5dc5b6: Deleting consensus metadata
I20260926 16:44:14.979645 17675 raft_consensus.cc:2258] T 951a8e6683de43219d8d400147ae1a01 P e7a44bfa35494003aadce77ec43974a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:44:14.981859 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 16316
I20260926 16:44:14.983050 16378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:14.984694 17674 ts_tablet_manager.cc:1905] T 951a8e6683de43219d8d400147ae1a01 P ee01f03feda544c59ff2dd4f5467404b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:44:14.986392 17675 ts_tablet_manager.cc:1905] T 951a8e6683de43219d8d400147ae1a01 P e7a44bfa35494003aadce77ec43974a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:44:14.992381 17674 ts_tablet_manager.cc:1918] T 951a8e6683de43219d8d400147ae1a01 P ee01f03feda544c59ff2dd4f5467404b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:44:14.992583 17674 log.cc:1196] T 951a8e6683de43219d8d400147ae1a01 P ee01f03feda544c59ff2dd4f5467404b: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/wal/wals/951a8e6683de43219d8d400147ae1a01
I20260926 16:44:14.993160 17674 ts_tablet_manager.cc:1939] T 951a8e6683de43219d8d400147ae1a01 P ee01f03feda544c59ff2dd4f5467404b: Deleting consensus metadata
I20260926 16:44:15.000734 17675 ts_tablet_manager.cc:1918] T 951a8e6683de43219d8d400147ae1a01 P e7a44bfa35494003aadce77ec43974a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:44:15.000918 17675 log.cc:1196] T 951a8e6683de43219d8d400147ae1a01 P e7a44bfa35494003aadce77ec43974a8: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/wal/wals/951a8e6683de43219d8d400147ae1a01
I20260926 16:44:15.001518 17675 ts_tablet_manager.cc:1939] T 951a8e6683de43219d8d400147ae1a01 P e7a44bfa35494003aadce77ec43974a8: Deleting consensus metadata
I20260926 16:44:15.009692 17617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:14.962292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52154 (local address 127.9.238.253:38171)
0926 16:44:14.962463 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:14.962473 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:14.985464 (+ 22991us) server_negotiation.cc:407] Connection header received
0926 16:44:14.986006 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:14.986012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:14.986113 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:14.986225 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:14.991944 (+ 5719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.993209 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.995585 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:14.996280 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:14.996407 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:15.001372 (+ 4965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:15.001383 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:15.001389 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:15.001459 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:15.005603 (+ 4144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:15.006854 (+ 1251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:15.006861 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:15.006866 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:15.007106 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:15.007675 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:15.007682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:15.007686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:15.008165 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:15.008502 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:15.008903 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 16:44:15.009233 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 16:44:15.028649 17618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:14.980983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60464 (local address 127.9.238.252:34171)
0926 16:44:14.982416 (+ 1433us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:14.982423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:15.009548 (+ 27125us) server_negotiation.cc:407] Connection header received
0926 16:44:15.009794 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:15.009800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:15.009904 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:15.009974 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:15.011902 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:15.013288 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:15.015247 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:15.016020 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:15.016196 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:15.020803 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:15.020812 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:15.020818 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:15.020872 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:15.025088 (+ 4216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:15.026229 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:15.026236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:15.026239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:15.026395 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:15.026895 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:15.026900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:15.026904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:15.027283 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:15.027533 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:15.028036 (+ 503us) server_negotiation.cc:299] Negotiation successful
0926 16:44:15.028278 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1282}
I20260926 16:44:15.117827 16334 catalog_manager.cc:4841] TS e7a44bfa35494003aadce77ec43974a8 (127.9.238.193:40747): tablet 951a8e6683de43219d8d400147ae1a01 (table test-table [id=a7f5394bfe9e4431b6c25b7b1170b3bb]) successfully deleted
I20260926 16:44:15.119704 16334 catalog_manager.cc:4841] TS ee01f03feda544c59ff2dd4f5467404b (127.9.238.195:42613): tablet 951a8e6683de43219d8d400147ae1a01 (table test-table [id=a7f5394bfe9e4431b6c25b7b1170b3bb]) successfully deleted
I20260926 16:44:15.119953 16335 catalog_manager.cc:4841] TS b9822a16ad8646d78ff6a980fd5dc5b6 (127.9.238.194:34685): tablet 951a8e6683de43219d8d400147ae1a01 (table test-table [id=a7f5394bfe9e4431b6c25b7b1170b3bb]) successfully deleted
I20260926 16:44:15.120568 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 16316
I20260926 16:44:15.170218 17618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:15.149173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60468 (local address 127.9.238.252:34171)
0926 16:44:15.149447 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:15.149455 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:15.152957 (+ 3502us) server_negotiation.cc:407] Connection header received
0926 16:44:15.153308 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:15.153314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:15.153416 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:15.153540 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:15.156682 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:15.157549 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:15.159173 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:15.159987 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:15.160123 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:15.163633 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:15.163639 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:15.163644 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:15.163692 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:15.167179 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:15.168105 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:15.168112 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:15.168116 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:15.168261 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:15.168733 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:15.168741 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:15.168747 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:15.169051 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:15.169210 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:15.169693 (+ 483us) server_negotiation.cc:299] Negotiation successful
0926 16:44:15.169902 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 16:44:15.172689 17617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:15.152994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52162 (local address 127.9.238.253:38171)
0926 16:44:15.153170 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:15.153175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:15.153441 (+ 266us) server_negotiation.cc:407] Connection header received
0926 16:44:15.153676 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:15.153683 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:15.153752 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:15.153848 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:15.156074 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:15.156978 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:15.159638 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:15.160351 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:15.160489 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:15.164706 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:15.164714 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:15.164722 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:15.164769 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:15.168615 (+ 3846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:15.169852 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:15.169861 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:15.169865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:15.170328 (+ 463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:15.170919 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:15.170926 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:15.170931 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:15.171319 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:15.171647 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:15.172232 (+ 585us) server_negotiation.cc:299] Negotiation successful
0926 16:44:15.172411 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 16:44:15.934860 17685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:15.933892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.9.238.194:33259)
0926 16:44:15.934506 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:15.934621 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":291,"threads_started":1}
W20260926 16:44:15.935336 17473 heartbeater.cc:644] Failed to heartbeat to 127.9.238.254:40511 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
I20260926 16:44:15.939739 17686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:15.938473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.9.238.195:55699)
0926 16:44:15.939322 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:15.939490 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":455,"threads_started":1}
W20260926 16:44:15.940222 17614 heartbeater.cc:644] Failed to heartbeat to 127.9.238.254:40511 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
I20260926 16:44:15.946794 17687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:15.945732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.9.238.193:51737)
0926 16:44:15.946270 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:15.946530 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":313,"threads_started":1}
W20260926 16:44:15.947427 17332 heartbeater.cc:644] Failed to heartbeat to 127.9.238.254:40511 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
I20260926 16:44:16.458175 17688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212c6cc5c6e6456b919eb5680dc91f11)
I20260926 16:44:16.458731 17688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: false } }
I20260926 16:44:16.459050 17689 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212c6cc5c6e6456b919eb5680dc91f11)
I20260926 16:44:16.459398 17689 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: false } }
I20260926 16:44:16.460960 17689 leader_election.cc:288] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212c6cc5c6e6456b919eb5680dc91f11 (127.9.238.254:40511), 83503b55cdba4750ad2e659cdf5f85d6 (127.9.238.252:34171)
I20260926 16:44:16.461058 17688 leader_election.cc:288] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212c6cc5c6e6456b919eb5680dc91f11 (127.9.238.254:40511), 8d007fde43b449489bda0be2c54c1587 (127.9.238.253:38171)
I20260926 16:44:16.464228 17692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.461538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.0.0.1:39462)
0926 16:44:16.463444 (+ 1906us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:16.463794 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1732,"thread_start_us":1547,"threads_started":1}
I20260926 16:44:16.464387 17690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.461318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.0.0.1:39456)
0926 16:44:16.463816 (+ 2498us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:16.464046 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2350,"thread_start_us":2126,"threads_started":1}
W20260926 16:44:16.464849 16941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
W20260926 16:44:16.466957 17132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
I20260926 16:44:16.473673 17691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.461324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:34171 (local address 127.0.0.1:60474)
0926 16:44:16.462119 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:16.462149 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:16.462432 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:16.463604 (+ 1172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:16.463608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:16.463633 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:16.464101 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:16.464117 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.466877 (+ 2760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.466886 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:16.470171 (+ 3285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:16.470193 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.472729 (+ 2536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.472735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:16.472906 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:16.472959 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 16:44:16.473038 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 16:44:16.473208 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":1,"thread_start_us":596,"threads_started":1}
I20260926 16:44:16.474509 17692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.474068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.0.0.1:39472)
0926 16:44:16.474232 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:16.474320 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 16:44:16.474812 16941 leader_election.cc:334] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 212c6cc5c6e6456b919eb5680dc91f11 (127.9.238.254:40511): Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
I20260926 16:44:16.475088 17693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.461678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60474 (local address 127.9.238.252:34171)
0926 16:44:16.463128 (+ 1450us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:16.463137 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:16.463162 (+ 25us) server_negotiation.cc:407] Connection header received
0926 16:44:16.463257 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:16.463265 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:16.463367 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:16.464281 (+ 914us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:16.464724 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.466402 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.470457 (+ 4055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.472488 (+ 2031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.474029 (+ 1541us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:16.474212 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:16.474298 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 16:44:16.474518 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1253,"thread_start_us":1062,"threads_started":1}
I20260926 16:44:16.475425 17694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.474660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.0.0.1:39478)
0926 16:44:16.474901 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:16.475136 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260926 16:44:16.475781 17132 leader_election.cc:334] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 212c6cc5c6e6456b919eb5680dc91f11 (127.9.238.254:40511): Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
I20260926 16:44:16.475978 17156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d007fde43b449489bda0be2c54c1587" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "83503b55cdba4750ad2e659cdf5f85d6" is_pre_election: true
I20260926 16:44:16.476565 17156 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d007fde43b449489bda0be2c54c1587 in term 1.
I20260926 16:44:16.477044 17690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.463900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:38171 (local address 127.0.0.1:52172)
0926 16:44:16.464662 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:16.464687 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:16.464911 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:16.465831 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:16.465842 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:16.465900 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:16.466517 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:16.466537 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.469429 (+ 2892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.469440 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:16.473260 (+ 3820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:16.473274 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.476106 (+ 2832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.476114 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:16.476309 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:16.476350 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 16:44:16.476477 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 16:44:16.476652 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"mutex_wait_us":619,"thread_start_us":2655,"threads_started":1}
I20260926 16:44:16.477247 17695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.464262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52172 (local address 127.9.238.253:38171)
0926 16:44:16.464927 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:16.464935 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:16.464969 (+ 34us) server_negotiation.cc:407] Connection header received
0926 16:44:16.465081 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:16.465089 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:16.465197 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:16.465336 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:16.466793 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.469158 (+ 2365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.474320 (+ 5162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.475683 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.475823 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:16.475934 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:16.476571 (+ 637us) server_negotiation.cc:299] Negotiation successful
0926 16:44:16.476790 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":275,"threads_started":1}
I20260926 16:44:16.477368 16939 leader_election.cc:302] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83503b55cdba4750ad2e659cdf5f85d6, 8d007fde43b449489bda0be2c54c1587; no voters: 212c6cc5c6e6456b919eb5680dc91f11
I20260926 16:44:16.477860 16965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "83503b55cdba4750ad2e659cdf5f85d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8d007fde43b449489bda0be2c54c1587" is_pre_election: true
I20260926 16:44:16.478056 17689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 16:44:16.478175 17689 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 212c6cc5c6e6456b919eb5680dc91f11)
I20260926 16:44:16.478276 17689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [term 1 FOLLOWER]: Advancing to term 2
I20260926 16:44:16.480660 17689 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: false } }
I20260926 16:44:16.481069 16965 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83503b55cdba4750ad2e659cdf5f85d6 in current term 2: Already voted for candidate 8d007fde43b449489bda0be2c54c1587 in this term.
I20260926 16:44:16.481446 17689 leader_election.cc:288] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [CANDIDATE]: Term 2 election: Requested vote from peers 212c6cc5c6e6456b919eb5680dc91f11 (127.9.238.254:40511), 83503b55cdba4750ad2e659cdf5f85d6 (127.9.238.252:34171)
I20260926 16:44:16.482117 17692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.481648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.0.0.1:39500)
0926 16:44:16.481862 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:16.481954 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260926 16:44:16.482124 17156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d007fde43b449489bda0be2c54c1587" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "83503b55cdba4750ad2e659cdf5f85d6"
I20260926 16:44:16.482280 17132 leader_election.cc:302] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83503b55cdba4750ad2e659cdf5f85d6; no voters: 212c6cc5c6e6456b919eb5680dc91f11, 8d007fde43b449489bda0be2c54c1587
I20260926 16:44:16.482470 17156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [term 1 FOLLOWER]: Advancing to term 2
I20260926 16:44:16.483179 17692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.482891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.0.0.1:39514)
0926 16:44:16.482999 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:16.483066 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 16:44:16.483444 16941 leader_election.cc:334] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 212c6cc5c6e6456b919eb5680dc91f11 (127.9.238.254:40511): Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
I20260926 16:44:16.485127 17156 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d007fde43b449489bda0be2c54c1587 in term 2.
I20260926 16:44:16.485395 17688 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 16:44:16.485764 16939 leader_election.cc:302] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83503b55cdba4750ad2e659cdf5f85d6, 8d007fde43b449489bda0be2c54c1587; no voters: 212c6cc5c6e6456b919eb5680dc91f11
I20260926 16:44:16.486181 17689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 16:44:16.486814 17689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [term 2 LEADER]: Becoming Leader. State: Replica: 8d007fde43b449489bda0be2c54c1587, State: Running, Role: LEADER
I20260926 16:44:16.487224 17689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: false } }
I20260926 16:44:16.489184 17698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d007fde43b449489bda0be2c54c1587. Latest consensus state: current_term: 2 leader_uuid: "8d007fde43b449489bda0be2c54c1587" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: false } } }
I20260926 16:44:16.489809 17698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:16.490768 17700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:44:16.494674 17700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:44:16.495499 17700 catalog_manager.cc:1259] Loaded cluster ID: 58607067b23c4cbca0a242e80249d2d2
I20260926 16:44:16.495630 17700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:44:16.496819 17700 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:44:16.497622 17700 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587: Loaded TSK: 0
I20260926 16:44:16.498543 17700 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:44:16.540760 16954 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:44:16.548570 16954 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:44:16.553757 17156 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [term 2 FOLLOWER]: Refusing update from remote peer 8d007fde43b449489bda0be2c54c1587: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 16:44:16.553864 17692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.553314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.0.0.1:39524)
0926 16:44:16.553534 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:16.553633 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 16:44:16.554777 17698 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 16:44:16.555481 17692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.554896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.0.0.1:39540)
0926 16:44:16.555187 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:16.555302 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260926 16:44:16.555763 16941 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 -> Peer 212c6cc5c6e6456b919eb5680dc91f11 (127.9.238.254:40511): Couldn't send request to peer 212c6cc5c6e6456b919eb5680dc91f11. Status: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 16:44:16.558398 17688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d007fde43b449489bda0be2c54c1587. Latest consensus state: current_term: 2 leader_uuid: "8d007fde43b449489bda0be2c54c1587" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: false } } }
I20260926 16:44:16.558754 17688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:44:16.561167 17689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "8d007fde43b449489bda0be2c54c1587" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: false } } }
I20260926 16:44:16.561622 17689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:16.562162 17688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "8d007fde43b449489bda0be2c54c1587" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "212c6cc5c6e6456b919eb5680dc91f11" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40511 } } peers { permanent_uuid: "8d007fde43b449489bda0be2c54c1587" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 38171 } attrs { promote: false } } peers { permanent_uuid: "83503b55cdba4750ad2e659cdf5f85d6" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 34171 } attrs { promote: false } } }
I20260926 16:44:16.562533 17688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83503b55cdba4750ad2e659cdf5f85d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:44:16.582258 17706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.572549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33610 (local address 127.9.238.194:34685)
0926 16:44:16.573132 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:16.573137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:16.573152 (+ 15us) server_negotiation.cc:407] Connection header received
0926 16:44:16.573209 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:16.573212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:16.573266 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:16.574188 (+ 922us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:16.574432 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.575980 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.579285 (+ 3305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.580757 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.581586 (+ 829us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:16.581708 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:16.581762 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 16:44:16.581899 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":337,"threads_started":1}
I20260926 16:44:16.582295 17692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.572345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:34685 (local address 127.0.0.1:33610)
0926 16:44:16.572520 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:16.572542 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:16.572651 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:16.573453 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:16.573458 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:16.573483 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:16.573979 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:16.573994 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.576196 (+ 2202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.576205 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:16.578979 (+ 2774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:16.579003 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.581078 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.581133 (+ 55us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:16.581580 (+ 447us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:16.581630 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 16:44:16.581735 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 16:44:16.581899 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":87}
I20260926 16:44:16.583688 17407 tablet_service.cc:1467] Processing CreateTablet for tablet 48c10b88cd194f7e95cceb54658c3fd8 (DEFAULT_TABLE table=test-table [id=ed065af467424d4cb5a81fe20ba3f9bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:16.584193 17407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48c10b88cd194f7e95cceb54658c3fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:16.588907 17691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.573290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:40747 (local address 127.0.0.1:44270)
0926 16:44:16.573496 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:16.573519 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:16.573640 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:16.581951 (+ 8311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:16.581956 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:16.581987 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:16.582182 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:16.582194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.583953 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.583988 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:16.586243 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:16.586262 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.588008 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.588015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:16.588261 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:16.588316 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 16:44:16.588394 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 16:44:16.588556 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":108}
I20260926 16:44:16.590152 17707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.573464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44270 (local address 127.9.238.193:40747)
0926 16:44:16.581529 (+ 8065us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:16.581536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:16.581557 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:44:16.581645 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:16.581653 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:16.581767 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:16.582334 (+ 567us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:16.582501 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.583749 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.586496 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.587802 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.589456 (+ 1654us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:16.589580 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:16.589644 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 16:44:16.589794 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7901,"thread_start_us":7767,"threads_started":1}
I20260926 16:44:16.591701 17266 tablet_service.cc:1467] Processing CreateTablet for tablet 48c10b88cd194f7e95cceb54658c3fd8 (DEFAULT_TABLE table=test-table [id=ed065af467424d4cb5a81fe20ba3f9bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:16.592111 17709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.575081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:42613 (local address 127.0.0.1:37394)
0926 16:44:16.575822 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:16.575850 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:16.576026 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:16.580919 (+ 4893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:16.580924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:16.580954 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:16.581544 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:16.581559 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.583410 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.583415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:16.587437 (+ 4022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:16.587469 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.590994 (+ 3525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.591001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:16.591201 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:16.591234 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 16:44:16.591354 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 16:44:16.591533 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":82,"thread_start_us":463,"threads_started":1}
I20260926 16:44:16.592226 17708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.573810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37394 (local address 127.9.238.195:42613)
0926 16:44:16.580531 (+ 6721us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:16.580537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:16.580558 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:44:16.580636 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:16.580640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:16.580736 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:16.580899 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:16.581788 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.583231 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.589257 (+ 6026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.590703 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.591049 (+ 346us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:16.591619 (+ 570us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:16.591705 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 16:44:16.591882 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6586,"thread_start_us":6450,"threads_started":1}
I20260926 16:44:16.592422 17266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48c10b88cd194f7e95cceb54658c3fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:16.593389 17548 tablet_service.cc:1467] Processing CreateTablet for tablet 48c10b88cd194f7e95cceb54658c3fd8 (DEFAULT_TABLE table=test-table [id=ed065af467424d4cb5a81fe20ba3f9bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:16.593925 17548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48c10b88cd194f7e95cceb54658c3fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:16.599562 17710 tablet_bootstrap.cc:492] T 48c10b88cd194f7e95cceb54658c3fd8 P b9822a16ad8646d78ff6a980fd5dc5b6: Bootstrap starting.
I20260926 16:44:16.600793 17711 tablet_bootstrap.cc:492] T 48c10b88cd194f7e95cceb54658c3fd8 P e7a44bfa35494003aadce77ec43974a8: Bootstrap starting.
I20260926 16:44:16.601850 17712 tablet_bootstrap.cc:492] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b: Bootstrap starting.
I20260926 16:44:16.602248 17710 tablet_bootstrap.cc:654] T 48c10b88cd194f7e95cceb54658c3fd8 P b9822a16ad8646d78ff6a980fd5dc5b6: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:16.602743 17711 tablet_bootstrap.cc:654] T 48c10b88cd194f7e95cceb54658c3fd8 P e7a44bfa35494003aadce77ec43974a8: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:16.604001 17712 tablet_bootstrap.cc:654] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:16.605336 17710 tablet_bootstrap.cc:492] T 48c10b88cd194f7e95cceb54658c3fd8 P b9822a16ad8646d78ff6a980fd5dc5b6: No bootstrap required, opened a new log
I20260926 16:44:16.605581 17710 ts_tablet_manager.cc:1397] T 48c10b88cd194f7e95cceb54658c3fd8 P b9822a16ad8646d78ff6a980fd5dc5b6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 16:44:16.606585 17711 tablet_bootstrap.cc:492] T 48c10b88cd194f7e95cceb54658c3fd8 P e7a44bfa35494003aadce77ec43974a8: No bootstrap required, opened a new log
I20260926 16:44:16.606564 17710 raft_consensus.cc:348] T 48c10b88cd194f7e95cceb54658c3fd8 P b9822a16ad8646d78ff6a980fd5dc5b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9822a16ad8646d78ff6a980fd5dc5b6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34685 } } peers { permanent_uuid: "e7a44bfa35494003aadce77ec43974a8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 40747 } } peers { permanent_uuid: "ee01f03feda544c59ff2dd4f5467404b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 42613 } }
I20260926 16:44:16.606825 17710 raft_consensus.cc:374] T 48c10b88cd194f7e95cceb54658c3fd8 P b9822a16ad8646d78ff6a980fd5dc5b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:16.606838 17711 ts_tablet_manager.cc:1397] T 48c10b88cd194f7e95cceb54658c3fd8 P e7a44bfa35494003aadce77ec43974a8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 16:44:16.606928 17710 raft_consensus.cc:729] T 48c10b88cd194f7e95cceb54658c3fd8 P b9822a16ad8646d78ff6a980fd5dc5b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9822a16ad8646d78ff6a980fd5dc5b6, State: Initialized, Role: FOLLOWER
I20260926 16:44:16.607263 17710 consensus_queue.cc:260] T 48c10b88cd194f7e95cceb54658c3fd8 P b9822a16ad8646d78ff6a980fd5dc5b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9822a16ad8646d78ff6a980fd5dc5b6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34685 } } peers { permanent_uuid: "e7a44bfa35494003aadce77ec43974a8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 40747 } } peers { permanent_uuid: "ee01f03feda544c59ff2dd4f5467404b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 42613 } }
I20260926 16:44:16.607843 17711 raft_consensus.cc:348] T 48c10b88cd194f7e95cceb54658c3fd8 P e7a44bfa35494003aadce77ec43974a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9822a16ad8646d78ff6a980fd5dc5b6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34685 } } peers { permanent_uuid: "e7a44bfa35494003aadce77ec43974a8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 40747 } } peers { permanent_uuid: "ee01f03feda544c59ff2dd4f5467404b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 42613 } }
I20260926 16:44:16.608139 17711 raft_consensus.cc:374] T 48c10b88cd194f7e95cceb54658c3fd8 P e7a44bfa35494003aadce77ec43974a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:16.608254 17711 raft_consensus.cc:729] T 48c10b88cd194f7e95cceb54658c3fd8 P e7a44bfa35494003aadce77ec43974a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a44bfa35494003aadce77ec43974a8, State: Initialized, Role: FOLLOWER
I20260926 16:44:16.608278 17710 ts_tablet_manager.cc:1428] T 48c10b88cd194f7e95cceb54658c3fd8 P b9822a16ad8646d78ff6a980fd5dc5b6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 16:44:16.608604 17711 consensus_queue.cc:260] T 48c10b88cd194f7e95cceb54658c3fd8 P e7a44bfa35494003aadce77ec43974a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9822a16ad8646d78ff6a980fd5dc5b6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34685 } } peers { permanent_uuid: "e7a44bfa35494003aadce77ec43974a8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 40747 } } peers { permanent_uuid: "ee01f03feda544c59ff2dd4f5467404b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 42613 } }
I20260926 16:44:16.608855 17712 tablet_bootstrap.cc:492] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b: No bootstrap required, opened a new log
I20260926 16:44:16.609037 17712 ts_tablet_manager.cc:1397] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 16:44:16.610054 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.608932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.9.238.194:44651)
0926 16:44:16.609600 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:16.609769 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"threads_started":1}
I20260926 16:44:16.610267 17712 raft_consensus.cc:348] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9822a16ad8646d78ff6a980fd5dc5b6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34685 } } peers { permanent_uuid: "e7a44bfa35494003aadce77ec43974a8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 40747 } } peers { permanent_uuid: "ee01f03feda544c59ff2dd4f5467404b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 42613 } }
I20260926 16:44:16.610556 17712 raft_consensus.cc:374] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:16.610694 17712 raft_consensus.cc:729] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee01f03feda544c59ff2dd4f5467404b, State: Initialized, Role: FOLLOWER
I20260926 16:44:16.611028 17712 consensus_queue.cc:260] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9822a16ad8646d78ff6a980fd5dc5b6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34685 } } peers { permanent_uuid: "e7a44bfa35494003aadce77ec43974a8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 40747 } } peers { permanent_uuid: "ee01f03feda544c59ff2dd4f5467404b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 42613 } }
I20260926 16:44:16.614589 17712 ts_tablet_manager.cc:1428] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 16:44:16.615989 17711 ts_tablet_manager.cc:1428] T 48c10b88cd194f7e95cceb54658c3fd8 P e7a44bfa35494003aadce77ec43974a8: Time spent starting tablet: real 0.009s user 0.004s sys 0.005s
I20260926 16:44:16.620954 17721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.616087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.9.238.193:49911)
0926 16:44:16.620437 (+ 4350us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:16.620655 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4199,"thread_start_us":4030,"threads_started":1}
I20260926 16:44:16.620963 17474 heartbeater.cc:503] Master 127.9.238.253:38171 was elected leader, sending a full tablet report...
I20260926 16:44:16.622272 17720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.613779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.9.238.195:50869)
0926 16:44:16.621807 (+ 8028us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:16.621982 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7855,"thread_start_us":7710,"threads_started":1}
I20260926 16:44:16.624086 17333 heartbeater.cc:503] Master 127.9.238.253:38171 was elected leader, sending a full tablet report...
I20260926 16:44:16.624691 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.621636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.9.238.194:40041)
0926 16:44:16.624415 (+ 2779us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:16.624520 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2653}
I20260926 16:44:16.625720 17615 heartbeater.cc:503] Master 127.9.238.253:38171 was elected leader, sending a full tablet report...
I20260926 16:44:16.626106 17721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.623531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.9.238.193:42073)
0926 16:44:16.625624 (+ 2093us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:16.625762 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1943}
I20260926 16:44:16.626467 17720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.625891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.9.238.195:51409)
0926 16:44:16.626209 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:16.626323 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
I20260926 16:44:16.643833 17719 raft_consensus.cc:482] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:44:16.644523 17719 raft_consensus.cc:504] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9822a16ad8646d78ff6a980fd5dc5b6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34685 } } peers { permanent_uuid: "e7a44bfa35494003aadce77ec43974a8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 40747 } } peers { permanent_uuid: "ee01f03feda544c59ff2dd4f5467404b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 42613 } }
I20260926 16:44:16.646701 17719 leader_election.cc:288] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9822a16ad8646d78ff6a980fd5dc5b6 (127.9.238.194:34685), e7a44bfa35494003aadce77ec43974a8 (127.9.238.193:40747)
I20260926 16:44:16.654030 17720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.647112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:40747 (local address 127.9.238.195:59977)
0926 16:44:16.647479 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:16.647508 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:16.648283 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:16.648959 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:16.648964 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:16.649024 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:16.649364 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:16.649376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.650426 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.650431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:16.652110 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:16.652124 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.653274 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.653280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:16.653485 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:16.653510 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 16:44:16.653634 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 16:44:16.653761 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":1,"spinlock_wait_cycles":5888}
I20260926 16:44:16.654587 17707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.647288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:59977 (local address 127.9.238.193:40747)
0926 16:44:16.647610 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:16.647638 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:16.647665 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:44:16.648538 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:16.648544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:16.648652 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:16.648726 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:16.649537 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.650274 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.652294 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.653077 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.653267 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:16.653346 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:16.653757 (+ 411us) server_negotiation.cc:299] Negotiation successful
0926 16:44:16.653972 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 16:44:16.655289 17286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c10b88cd194f7e95cceb54658c3fd8" candidate_uuid: "ee01f03feda544c59ff2dd4f5467404b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a44bfa35494003aadce77ec43974a8" is_pre_election: true
I20260926 16:44:16.655582 17706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.647375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:55783 (local address 127.9.238.194:34685)
0926 16:44:16.647612 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:16.647620 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:16.648350 (+ 730us) server_negotiation.cc:407] Connection header received
0926 16:44:16.648703 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:16.648711 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:16.648800 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:16.648902 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:16.649704 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.650476 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.652693 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.654001 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.654242 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:16.654332 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:16.655054 (+ 722us) server_negotiation.cc:299] Negotiation successful
0926 16:44:16.655224 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 16:44:16.655618 17286 raft_consensus.cc:2454] T 48c10b88cd194f7e95cceb54658c3fd8 P e7a44bfa35494003aadce77ec43974a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee01f03feda544c59ff2dd4f5467404b in term 0.
I20260926 16:44:16.655714 17722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.647112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:34685 (local address 127.9.238.195:55783)
0926 16:44:16.648180 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:16.648202 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:16.648488 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:16.648959 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:16.648964 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:16.649007 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:16.649518 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:16.649528 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.650629 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.650636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:16.652476 (+ 1840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:16.652491 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.654533 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.654544 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:16.654818 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:16.654866 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 16:44:16.655022 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 16:44:16.655199 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"mutex_wait_us":65,"spinlock_wait_cycles":4992,"thread_start_us":645,"threads_started":1}
I20260926 16:44:16.656263 17503 leader_election.cc:302] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e7a44bfa35494003aadce77ec43974a8, ee01f03feda544c59ff2dd4f5467404b; no voters:
I20260926 16:44:16.656723 17719 raft_consensus.cc:2781] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:44:16.656903 17719 raft_consensus.cc:482] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:44:16.657033 17719 raft_consensus.cc:3037] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:16.656906 17427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c10b88cd194f7e95cceb54658c3fd8" candidate_uuid: "ee01f03feda544c59ff2dd4f5467404b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9822a16ad8646d78ff6a980fd5dc5b6" is_pre_election: true
I20260926 16:44:16.657449 17427 raft_consensus.cc:2454] T 48c10b88cd194f7e95cceb54658c3fd8 P b9822a16ad8646d78ff6a980fd5dc5b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee01f03feda544c59ff2dd4f5467404b in term 0.
I20260926 16:44:16.660033 17719 raft_consensus.cc:504] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9822a16ad8646d78ff6a980fd5dc5b6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34685 } } peers { permanent_uuid: "e7a44bfa35494003aadce77ec43974a8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 40747 } } peers { permanent_uuid: "ee01f03feda544c59ff2dd4f5467404b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 42613 } }
I20260926 16:44:16.660866 17719 leader_election.cc:288] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b [CANDIDATE]: Term 1 election: Requested vote from peers b9822a16ad8646d78ff6a980fd5dc5b6 (127.9.238.194:34685), e7a44bfa35494003aadce77ec43974a8 (127.9.238.193:40747)
I20260926 16:44:16.661540 17427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c10b88cd194f7e95cceb54658c3fd8" candidate_uuid: "ee01f03feda544c59ff2dd4f5467404b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9822a16ad8646d78ff6a980fd5dc5b6"
I20260926 16:44:16.661672 17286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c10b88cd194f7e95cceb54658c3fd8" candidate_uuid: "ee01f03feda544c59ff2dd4f5467404b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7a44bfa35494003aadce77ec43974a8"
I20260926 16:44:16.661895 17427 raft_consensus.cc:3037] T 48c10b88cd194f7e95cceb54658c3fd8 P b9822a16ad8646d78ff6a980fd5dc5b6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:16.661947 17286 raft_consensus.cc:3037] T 48c10b88cd194f7e95cceb54658c3fd8 P e7a44bfa35494003aadce77ec43974a8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:16.665894 17286 raft_consensus.cc:2454] T 48c10b88cd194f7e95cceb54658c3fd8 P e7a44bfa35494003aadce77ec43974a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee01f03feda544c59ff2dd4f5467404b in term 1.
I20260926 16:44:16.665894 17427 raft_consensus.cc:2454] T 48c10b88cd194f7e95cceb54658c3fd8 P b9822a16ad8646d78ff6a980fd5dc5b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee01f03feda544c59ff2dd4f5467404b in term 1.
I20260926 16:44:16.666589 17504 leader_election.cc:302] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9822a16ad8646d78ff6a980fd5dc5b6, ee01f03feda544c59ff2dd4f5467404b; no voters:
I20260926 16:44:16.667057 17719 raft_consensus.cc:2781] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:44:16.667747 17719 raft_consensus.cc:686] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b [term 1 LEADER]: Becoming Leader. State: Replica: ee01f03feda544c59ff2dd4f5467404b, State: Running, Role: LEADER
W20260926 16:44:16.667900 17475 tablet.cc:2369] T 48c10b88cd194f7e95cceb54658c3fd8 P b9822a16ad8646d78ff6a980fd5dc5b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:44:16.668203 17719 consensus_queue.cc:237] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9822a16ad8646d78ff6a980fd5dc5b6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34685 } } peers { permanent_uuid: "e7a44bfa35494003aadce77ec43974a8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 40747 } } peers { permanent_uuid: "ee01f03feda544c59ff2dd4f5467404b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 42613 } }
I20260926 16:44:16.669656 17722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.668982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.9.238.195:57965)
0926 16:44:16.669227 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:16.669451 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 16:44:16.672289 16953 catalog_manager.cc:5495] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b reported cstate change: term changed from 0 to 1, leader changed from <none> to ee01f03feda544c59ff2dd4f5467404b (127.9.238.195). New cstate: current_term: 1 leader_uuid: "ee01f03feda544c59ff2dd4f5467404b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9822a16ad8646d78ff6a980fd5dc5b6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7a44bfa35494003aadce77ec43974a8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 40747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee01f03feda544c59ff2dd4f5467404b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 42613 } health_report { overall_health: HEALTHY } } }
I20260926 16:44:16.700227 17708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.693698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37406 (local address 127.9.238.195:42613)
0926 16:44:16.693972 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:16.693979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:16.694004 (+ 25us) server_negotiation.cc:407] Connection header received
0926 16:44:16.694197 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:16.694202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:16.694328 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:16.694525 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:44:16.694969 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.695962 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.697520 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.698010 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.698168 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:16.698668 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:44:16.699464 (+ 796us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:44:16.699704 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:16.699767 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 16:44:16.699870 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 16:44:16.705572 17286 raft_consensus.cc:1261] T 48c10b88cd194f7e95cceb54658c3fd8 P e7a44bfa35494003aadce77ec43974a8 [term 1 FOLLOWER]: Refusing update from remote peer ee01f03feda544c59ff2dd4f5467404b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:44:16.706182 17427 raft_consensus.cc:1261] T 48c10b88cd194f7e95cceb54658c3fd8 P b9822a16ad8646d78ff6a980fd5dc5b6 [term 1 FOLLOWER]: Refusing update from remote peer ee01f03feda544c59ff2dd4f5467404b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:44:16.706458 17723 consensus_queue.cc:1035] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7a44bfa35494003aadce77ec43974a8" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 40747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:44:16.707216 17719 consensus_queue.cc:1035] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9822a16ad8646d78ff6a980fd5dc5b6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:44:16.709368 17721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.708808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.9.238.193:35983)
0926 16:44:16.709005 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:16.709135 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 16:44:16.710427 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.709816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.9.238.194:55703)
0926 16:44:16.710052 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:16.710176 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 16:44:16.718314 17718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.717736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.9.238.194:55421)
0926 16:44:16.717979 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:16.718093 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260926 16:44:16.722039 17722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.719683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.9.238.195:37987)
0926 16:44:16.721623 (+ 1940us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:16.721759 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1806}
I20260926 16:44:16.726181 17721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.725578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.9.238.193:55707)
0926 16:44:16.725826 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:16.725949 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260926 16:44:16.726922 17722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.725598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.9.238.195:54451)
0926 16:44:16.726137 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:16.726704 (+ 567us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428}
I20260926 16:44:16.728425 17729 mvcc.cc:205] Tried to move back new op lower bound from 7333646568256765952 to 7333646568117350400. Current Snapshot: MvccSnapshot[applied={T|T < 7333646568256765952}]
I20260926 16:44:16.747099 16953 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:44:16.747440 16953 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:44:16.756484 16953 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d007fde43b449489bda0be2c54c1587: Sending DeleteTablet for 3 replicas of tablet 48c10b88cd194f7e95cceb54658c3fd8
I20260926 16:44:16.757721 17407 tablet_service.cc:1514] Processing DeleteTablet for tablet 48c10b88cd194f7e95cceb54658c3fd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:16 UTC) from {username='oryx', principal='oryx/127.9.238.253@KRBTEST.COM'} at 127.0.0.1:33610
I20260926 16:44:16.758111 17266 tablet_service.cc:1514] Processing DeleteTablet for tablet 48c10b88cd194f7e95cceb54658c3fd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:16 UTC) from {username='oryx', principal='oryx/127.9.238.253@KRBTEST.COM'} at 127.0.0.1:44270
I20260926 16:44:16.758473 17548 tablet_service.cc:1514] Processing DeleteTablet for tablet 48c10b88cd194f7e95cceb54658c3fd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:16 UTC) from {username='oryx', principal='oryx/127.9.238.253@KRBTEST.COM'} at 127.0.0.1:37394
I20260926 16:44:16.764871 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 17118
I20260926 16:44:16.770545 17736 tablet_replica.cc:330] T 48c10b88cd194f7e95cceb54658c3fd8 P b9822a16ad8646d78ff6a980fd5dc5b6: stopping tablet replica
I20260926 16:44:16.770835 17736 raft_consensus.cc:2229] T 48c10b88cd194f7e95cceb54658c3fd8 P b9822a16ad8646d78ff6a980fd5dc5b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:44:16.771237 17736 raft_consensus.cc:2258] T 48c10b88cd194f7e95cceb54658c3fd8 P b9822a16ad8646d78ff6a980fd5dc5b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:44:16.772024 17737 tablet_replica.cc:330] T 48c10b88cd194f7e95cceb54658c3fd8 P e7a44bfa35494003aadce77ec43974a8: stopping tablet replica
I20260926 16:44:16.772274 17737 raft_consensus.cc:2229] T 48c10b88cd194f7e95cceb54658c3fd8 P e7a44bfa35494003aadce77ec43974a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:44:16.772663 17737 raft_consensus.cc:2258] T 48c10b88cd194f7e95cceb54658c3fd8 P e7a44bfa35494003aadce77ec43974a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:44:16.773669 17736 ts_tablet_manager.cc:1905] T 48c10b88cd194f7e95cceb54658c3fd8 P b9822a16ad8646d78ff6a980fd5dc5b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:44:16.773675 17737 ts_tablet_manager.cc:1905] T 48c10b88cd194f7e95cceb54658c3fd8 P e7a44bfa35494003aadce77ec43974a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:44:16.773698 17738 tablet_replica.cc:330] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b: stopping tablet replica
I20260926 16:44:16.773939 17738 raft_consensus.cc:2229] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:44:16.776553 17738 raft_consensus.cc:2258] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:44:16.780320 17738 ts_tablet_manager.cc:1905] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:44:16.781581 17736 ts_tablet_manager.cc:1918] T 48c10b88cd194f7e95cceb54658c3fd8 P b9822a16ad8646d78ff6a980fd5dc5b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:44:16.781581 17737 ts_tablet_manager.cc:1918] T 48c10b88cd194f7e95cceb54658c3fd8 P e7a44bfa35494003aadce77ec43974a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:44:16.781750 17736 log.cc:1196] T 48c10b88cd194f7e95cceb54658c3fd8 P b9822a16ad8646d78ff6a980fd5dc5b6: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-1/wal/wals/48c10b88cd194f7e95cceb54658c3fd8
I20260926 16:44:16.781755 17737 log.cc:1196] T 48c10b88cd194f7e95cceb54658c3fd8 P e7a44bfa35494003aadce77ec43974a8: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-0/wal/wals/48c10b88cd194f7e95cceb54658c3fd8
I20260926 16:44:16.782265 17737 ts_tablet_manager.cc:1939] T 48c10b88cd194f7e95cceb54658c3fd8 P e7a44bfa35494003aadce77ec43974a8: Deleting consensus metadata
I20260926 16:44:16.782274 17736 ts_tablet_manager.cc:1939] T 48c10b88cd194f7e95cceb54658c3fd8 P b9822a16ad8646d78ff6a980fd5dc5b6: Deleting consensus metadata
I20260926 16:44:16.783761 16941 catalog_manager.cc:4841] TS e7a44bfa35494003aadce77ec43974a8 (127.9.238.193:40747): tablet 48c10b88cd194f7e95cceb54658c3fd8 (table test-table [id=ed065af467424d4cb5a81fe20ba3f9bb]) successfully deleted
I20260926 16:44:16.784170 16942 catalog_manager.cc:4841] TS b9822a16ad8646d78ff6a980fd5dc5b6 (127.9.238.194:34685): tablet 48c10b88cd194f7e95cceb54658c3fd8 (table test-table [id=ed065af467424d4cb5a81fe20ba3f9bb]) successfully deleted
I20260926 16:44:16.790897 17738 ts_tablet_manager.cc:1918] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:44:16.791109 17738 log.cc:1196] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790440988577176-10171-0/minicluster-data/ts-2/wal/wals/48c10b88cd194f7e95cceb54658c3fd8
I20260926 16:44:16.791735 17738 ts_tablet_manager.cc:1939] T 48c10b88cd194f7e95cceb54658c3fd8 P ee01f03feda544c59ff2dd4f5467404b: Deleting consensus metadata
I20260926 16:44:16.793450 16941 catalog_manager.cc:4841] TS ee01f03feda544c59ff2dd4f5467404b (127.9.238.195:42613): tablet 48c10b88cd194f7e95cceb54658c3fd8 (table test-table [id=ed065af467424d4cb5a81fe20ba3f9bb]) successfully deleted
I20260926 16:44:16.802281 17707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:16.767047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44274 (local address 127.9.238.193:40747)
0926 16:44:16.767352 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:16.767362 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:16.771163 (+ 3801us) server_negotiation.cc:407] Connection header received
0926 16:44:16.771603 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:16.771612 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:16.771770 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:16.771921 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:16.773651 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.774632 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.776000 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:16.776857 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:16.776980 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:16.793124 (+ 16144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:16.793152 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:16.793182 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:16.793246 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:16.797912 (+ 4666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:16.799197 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:16.799210 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:16.799214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:16.799416 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:16.799970 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:16.799976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:16.799980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:16.800526 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:16.800884 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:16.801244 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 16:44:16.801555 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 16:44:16.803337 17326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:16.976820 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 17118
I20260926 16:44:17.007236 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 17343
I20260926 16:44:17.030092 17706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:17.008320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33624 (local address 127.9.238.194:34685)
0926 16:44:17.008612 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:17.008621 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:17.008720 (+ 99us) server_negotiation.cc:407] Connection header received
0926 16:44:17.008970 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:17.008981 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:17.009094 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:17.009261 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:17.011165 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:17.012910 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:17.014883 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:17.015779 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:17.015946 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:17.020100 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:17.020126 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:17.020153 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:17.020237 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:17.025051 (+ 4814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:17.026621 (+ 1570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:17.026635 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:17.026641 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:17.026874 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:17.027584 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:17.027592 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:17.027596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:17.028260 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:17.028628 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:17.029070 (+ 442us) server_negotiation.cc:299] Negotiation successful
0926 16:44:17.029402 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260926 16:44:17.030973 17467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:17.150493 17742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:17.134714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.0.0.1:39552)
0926 16:44:17.150090 (+ 15376us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:17.150232 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15189,"threads_started":1}
I20260926 16:44:17.169212 17742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:17.168671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40511 (local address 127.0.0.1:39568)
0926 16:44:17.168888 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:17.169007 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 16:44:17.248211 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 17343
I20260926 16:44:17.273860 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 17485
I20260926 16:44:17.293102 17744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:17.274768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37410 (local address 127.9.238.195:42613)
0926 16:44:17.275395 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:17.275399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:17.275417 (+ 18us) server_negotiation.cc:407] Connection header received
0926 16:44:17.275508 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:17.275512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:17.275677 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:17.275826 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:17.277517 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:17.278775 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:17.280672 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:17.281423 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:17.281599 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:17.284400 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:17.284429 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:17.284458 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:17.284517 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:17.289409 (+ 4892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:17.290279 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:17.290290 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:17.290294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:17.290533 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:17.290958 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:17.290966 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:17.290972 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:17.291598 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:17.291953 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:17.292391 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 16:44:17.292683 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":319,"threads_started":1}
I20260926 16:44:17.294310 17608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:17.429085 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 17485
I20260926 16:44:17.452136 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 16811
I20260926 16:44:17.453744 16985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:17.580682 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 16811
I20260926 16:44:17.602748 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 16911
I20260926 16:44:17.604089 17176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:17.722398 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 16911
2026-09-26T16:44:17Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10638 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:44:17 dist-test-slave-wt2j krb5kdc[17750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:44:17 dist-test-slave-wt2j krb5kdc[17750](info): set up 2 sockets
Sep 26 16:44:17 dist-test-slave-wt2j krb5kdc[17750](info): commencing operation
krb5kdc: starting...
W20260926 16:44:18.877851 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.006s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:44:19 dist-test-slave-wt2j krb5kdc[17750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:44:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:44:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.254.keytab.
Entry for principal oryx/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.254.keytab.
I20260926 16:44:19.148900 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:41549
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.254.keytab
--principal=oryx/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:44835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:41549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:19.484117 17763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:19.484491 17763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:19.484665 17763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:19.492056 17763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:44:19.492234 17763 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 16:44:19.492311 17763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:19.492435 17763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:19.492524 17763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:44:19.492643 17763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:44:19.503322 17763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:44835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:41549
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.254.keytab
--principal=oryx/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:41549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:19.504783 17763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:19.506143 17763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:19.520984 17771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:19.521135 17768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:19.521135 17769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:19.521463 17763 server_base.cc:1034] running on GCE node
I20260926 16:44:19.522511 17763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:19.524096 17763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:19.525367 17763 hybrid_clock.cc:648] HybridClock initialized: now 1790441059525266 us; error 65 us; skew 500 ppm
I20260926 16:44:19.525776 17763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:19 dist-test-slave-wt2j krb5kdc[17750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441059, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:19.537478 17763 init.cc:377] Logged in from keytab as oryx/127.9.238.254@KRBTEST.COM (short username oryx)
I20260926 16:44:19.539641 17763 webserver.cc:460] Webserver started at http://127.9.238.254:36575/ using document root <none> and password file <none>
I20260926 16:44:19.540524 17763 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:19.540670 17763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:19.541044 17763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:19.544493 17763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "fcfbc557bbb54548b890bd7cc48574b4"
format_stamp: "Formatted at 2026-09-26 16:44:19 on dist-test-slave-wt2j"
server_key: "c19eb8bb3ad01ef0dc4df0c3df5563e0"
server_key_iv: "7d10ed06a3c607e4545387bb12fb6007"
server_key_version: "encryptionkey@0"
I20260926 16:44:19.545462 17763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "fcfbc557bbb54548b890bd7cc48574b4"
format_stamp: "Formatted at 2026-09-26 16:44:19 on dist-test-slave-wt2j"
server_key: "c19eb8bb3ad01ef0dc4df0c3df5563e0"
server_key_iv: "7d10ed06a3c607e4545387bb12fb6007"
server_key_version: "encryptionkey@0"
I20260926 16:44:19.551944 17763 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260926 16:44:19.556115 17778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:19.557842 17763 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 16:44:19.558066 17763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "fcfbc557bbb54548b890bd7cc48574b4"
format_stamp: "Formatted at 2026-09-26 16:44:19 on dist-test-slave-wt2j"
server_key: "c19eb8bb3ad01ef0dc4df0c3df5563e0"
server_key_iv: "7d10ed06a3c607e4545387bb12fb6007"
server_key_version: "encryptionkey@0"
I20260926 16:44:19.558303 17763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:19.574992 17763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:19.580881 17763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:19.581185 17763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:19.607793 17763 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:41549
I20260926 16:44:19.607818 17839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:41549 every 8 connection(s)
I20260926 16:44:19.609629 17763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:44:19.613986 17840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:19.617471 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 17763
I20260926 16:44:19.617722 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:44:19.618489 10171 external_mini_cluster.cc:1428] Setting key ebb4929110fa34daf667dae9f57f49ca
I20260926 16:44:19.626982 17840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcfbc557bbb54548b890bd7cc48574b4: Bootstrap starting.
I20260926 16:44:19.630887 17840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcfbc557bbb54548b890bd7cc48574b4: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:19.632104 17840 log.cc:824] T 00000000000000000000000000000000 P fcfbc557bbb54548b890bd7cc48574b4: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:19.635763 17840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcfbc557bbb54548b890bd7cc48574b4: No bootstrap required, opened a new log
Sep 26 16:44:19 dist-test-slave-wt2j krb5kdc[17750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM
I20260926 16:44:19.644804 17840 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcfbc557bbb54548b890bd7cc48574b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfbc557bbb54548b890bd7cc48574b4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 41549 } }
I20260926 16:44:19.645210 17840 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcfbc557bbb54548b890bd7cc48574b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:19.645359 17840 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcfbc557bbb54548b890bd7cc48574b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcfbc557bbb54548b890bd7cc48574b4, State: Initialized, Role: FOLLOWER
I20260926 16:44:19.646065 17840 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcfbc557bbb54548b890bd7cc48574b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfbc557bbb54548b890bd7cc48574b4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 41549 } }
I20260926 16:44:19.646335 17840 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcfbc557bbb54548b890bd7cc48574b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:44:19.646507 17840 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcfbc557bbb54548b890bd7cc48574b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:44:19.646742 17840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcfbc557bbb54548b890bd7cc48574b4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:19.648979 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:19.621178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49514 (local address 127.9.238.254:41549)
0926 16:44:19.622220 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:19.622242 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:19.622308 (+ 66us) server_negotiation.cc:407] Connection header received
0926 16:44:19.625104 (+ 2796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:19.625138 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:19.625719 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:19.626195 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:19.628055 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:19.630097 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:19.631750 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:19.632916 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:19.633096 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:19.638978 (+ 5882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:19.638990 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:19.639016 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:19.639076 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:19.643259 (+ 4183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:19.644344 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:19.644354 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:19.644363 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:19.644577 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:19.645008 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:19.645013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:19.645015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:19.645922 (+ 907us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:19.646293 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:19.646924 (+ 631us) server_negotiation.cc:299] Negotiation successful
0926 16:44:19.647494 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":473,"threads_started":1}
I20260926 16:44:19.649960 17840 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcfbc557bbb54548b890bd7cc48574b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfbc557bbb54548b890bd7cc48574b4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 41549 } }
I20260926 16:44:19.650514 17840 leader_election.cc:302] T 00000000000000000000000000000000 P fcfbc557bbb54548b890bd7cc48574b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcfbc557bbb54548b890bd7cc48574b4; no voters:
I20260926 16:44:19.651228 17840 leader_election.cc:288] T 00000000000000000000000000000000 P fcfbc557bbb54548b890bd7cc48574b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:44:19.651489 17845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcfbc557bbb54548b890bd7cc48574b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:44:19.652866 17845 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcfbc557bbb54548b890bd7cc48574b4 [term 1 LEADER]: Becoming Leader. State: Replica: fcfbc557bbb54548b890bd7cc48574b4, State: Running, Role: LEADER
I20260926 16:44:19.653721 17845 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcfbc557bbb54548b890bd7cc48574b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfbc557bbb54548b890bd7cc48574b4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 41549 } }
I20260926 16:44:19.653959 17840 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcfbc557bbb54548b890bd7cc48574b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:44:19.658058 17846 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcfbc557bbb54548b890bd7cc48574b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcfbc557bbb54548b890bd7cc48574b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfbc557bbb54548b890bd7cc48574b4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 41549 } } }
I20260926 16:44:19.658622 17846 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcfbc557bbb54548b890bd7cc48574b4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:19.659363 17847 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcfbc557bbb54548b890bd7cc48574b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcfbc557bbb54548b890bd7cc48574b4. Latest consensus state: current_term: 1 leader_uuid: "fcfbc557bbb54548b890bd7cc48574b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfbc557bbb54548b890bd7cc48574b4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 41549 } } }
I20260926 16:44:19.659765 17847 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcfbc557bbb54548b890bd7cc48574b4 [sys.catalog]: This master's current role is: LEADER
W20260926 16:44:19.671094 17860 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fcfbc557bbb54548b890bd7cc48574b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:44:19.671262 17860 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:44:19.671380 17855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:44:19.673650 17855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:44:19.685158 17855 catalog_manager.cc:1347] Generated new cluster ID: 05509a5eef3a4da7887db4e005f644e1
I20260926 16:44:19.685343 17855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:44:19.717909 17855 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:44:19.719725 17855 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:44:19.729216 17855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcfbc557bbb54548b890bd7cc48574b4: Generated new TSK 0
I20260926 16:44:19.730510 17855 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.9.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.193.keytab.
Entry for principal oryx/127.9.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.193.keytab.
I20260926 16:44:19.808789 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:0
--local_ip_for_outbound_sockets=127.9.238.193
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:41549
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.193.keytab
--principal=oryx/127.9.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:44835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:20.238577 17866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:20.238979 17866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:20.239212 17866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:20.247613 17866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:20.247790 17866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:20.248019 17866 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:44:20.248157 17866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:44:20.261571 17866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:44835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.193.keytab
--principal=oryx/127.9.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.238.254:41549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:20.263587 17866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:20.265362 17866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:20.284221 17874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:20.284348 17872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:20.284358 17871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:20.285396 17866 server_base.cc:1034] running on GCE node
I20260926 16:44:20.286238 17866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:20.287328 17866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:20.288664 17866 hybrid_clock.cc:648] HybridClock initialized: now 1790441060288551 us; error 86 us; skew 500 ppm
I20260926 16:44:20.288985 17866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:20 dist-test-slave-wt2j krb5kdc[17750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441060, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:20.292795 17866 init.cc:377] Logged in from keytab as oryx/127.9.238.193@KRBTEST.COM (short username oryx)
I20260926 16:44:20.294627 17866 webserver.cc:460] Webserver started at http://127.9.238.193:40885/ using document root <none> and password file <none>
I20260926 16:44:20.295454 17866 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:20.295606 17866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:20.295979 17866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:20.299206 17866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-0/data/instance:
uuid: "09bb826eee674015b88848b1c2e9c54d"
format_stamp: "Formatted at 2026-09-26 16:44:20 on dist-test-slave-wt2j"
server_key: "f5a1cae9e68999ffe998caa1cd0cc215"
server_key_iv: "2ad0cc7e8ddf23e1da5cd1a8ddc242e9"
server_key_version: "encryptionkey@0"
I20260926 16:44:20.300125 17866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance:
uuid: "09bb826eee674015b88848b1c2e9c54d"
format_stamp: "Formatted at 2026-09-26 16:44:20 on dist-test-slave-wt2j"
server_key: "f5a1cae9e68999ffe998caa1cd0cc215"
server_key_iv: "2ad0cc7e8ddf23e1da5cd1a8ddc242e9"
server_key_version: "encryptionkey@0"
I20260926 16:44:20.306093 17866 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 16:44:20.310415 17881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:20.312397 17866 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 16:44:20.312636 17866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "09bb826eee674015b88848b1c2e9c54d"
format_stamp: "Formatted at 2026-09-26 16:44:20 on dist-test-slave-wt2j"
server_key: "f5a1cae9e68999ffe998caa1cd0cc215"
server_key_iv: "2ad0cc7e8ddf23e1da5cd1a8ddc242e9"
server_key_version: "encryptionkey@0"
I20260926 16:44:20.312885 17866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:20.342526 17866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:20.347275 17866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:20.347620 17866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:20.350807 17866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:20.350965 17866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:20.351159 17866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:20.351238 17866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:44:20 dist-test-slave-wt2j krb5kdc[17750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441060, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM
I20260926 16:44:20.380694 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:20.351942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:37017 (local address 127.9.238.254:41549)
0926 16:44:20.352549 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:20.352557 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:20.354291 (+ 1734us) server_negotiation.cc:407] Connection header received
0926 16:44:20.358483 (+ 4192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:20.358492 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:20.358627 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:20.358857 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:20.361974 (+ 3117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:20.362866 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:20.365496 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:20.366123 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:20.366263 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:20.371422 (+ 5159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:20.371429 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:20.371432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:20.371472 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:20.376049 (+ 4577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:20.377106 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:20.377114 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:20.377121 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:20.377348 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:20.377951 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:20.377958 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:20.377964 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:20.378455 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:20.378670 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:20.379794 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0926 16:44:20.380122 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":294,"threads_started":1}
I20260926 16:44:20.381706 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:20.352887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:41549 (local address 127.9.238.193:37017)
0926 16:44:20.354009 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:20.354101 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:20.357945 (+ 3844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:20.359302 (+ 1357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:20.359321 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:20.360418 (+ 1097us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:20.361720 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:20.361749 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:20.363062 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:20.363068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:20.365260 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:20.365284 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:20.366396 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:20.366404 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:20.366689 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:20.367979 (+ 1290us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:20.368006 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:20.371088 (+ 3082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:20.376303 (+ 5215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:20.376320 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:20.376348 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:20.376911 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:20.377512 (+ 601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:20.377516 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:20.377519 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:20.377788 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:20.378656 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:20.378664 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:20.379191 (+ 527us) client_negotiation.cc:769] Sending connection context
0926 16:44:20.379633 (+ 442us) client_negotiation.cc:240] Negotiation successful
0926 16:44:20.380183 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"thread_start_us":474,"threads_started":1}
I20260926 16:44:20.407307 17948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:20.391018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:41549 (local address 127.9.238.193:33287)
0926 16:44:20.391913 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:20.391945 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:20.392203 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:20.392611 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:20.392616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:20.393378 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:20.393759 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:20.393770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:20.395007 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:20.395014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:20.396586 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:20.396602 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:20.397517 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:20.397525 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:20.397750 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:20.398916 (+ 1166us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:20.398965 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:20.400525 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:20.404007 (+ 3482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:20.404014 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:20.404019 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:20.404501 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:20.404999 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:20.405004 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:20.405009 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:20.405149 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:20.405995 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:20.406000 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:20.406245 (+ 245us) client_negotiation.cc:769] Sending connection context
0926 16:44:20.406476 (+ 231us) client_negotiation.cc:240] Negotiation successful
0926 16:44:20.406900 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":467,"threads_started":1}
I20260926 16:44:20.407807 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:20.391074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:33287 (local address 127.9.238.254:41549)
0926 16:44:20.391380 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:20.391390 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:20.392090 (+ 700us) server_negotiation.cc:407] Connection header received
0926 16:44:20.392371 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:20.392379 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:20.392462 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:20.392578 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:20.393958 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:20.394835 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:20.396768 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:20.397259 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:20.397411 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:20.400813 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:20.400819 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:20.400822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:20.400862 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:20.403805 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:20.404643 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:20.404651 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:20.404656 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:20.404834 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:20.405315 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:20.405322 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:20.405327 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:20.405829 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:20.406049 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:20.406794 (+ 745us) server_negotiation.cc:299] Negotiation successful
0926 16:44:20.407226 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 16:44:20.421464 17866 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:41345
I20260926 16:44:20.421502 18001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:41345 every 8 connection(s)
I20260926 16:44:20.423147 17866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:44:20.426295 18002 heartbeater.cc:347] Connected to a master server at 127.9.238.254:41549
I20260926 16:44:20.426568 18002 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:20.427253 18002 heartbeater.cc:511] Master 127.9.238.254:41549 requested a full tablet report, sending...
I20260926 16:44:20.429796 17795 ts_manager.cc:194] Registered new tserver with Master: 09bb826eee674015b88848b1c2e9c54d (127.9.238.193:41345)
I20260926 16:44:20.430327 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 17866
I20260926 16:44:20.430546 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance
I20260926 16:44:20.431376 10171 external_mini_cluster.cc:1428] Setting key df8be0c3cca3b3d5c3b2e08be726e83f
I20260926 16:44:20.432149 17795 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.238.193@KRBTEST.COM'} at 127.9.238.193:37017
WARNING: no policy specified for oryx/127.9.238.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.194.keytab.
Entry for principal oryx/127.9.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.194.keytab.
I20260926 16:44:20.493415 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:0
--local_ip_for_outbound_sockets=127.9.238.194
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:41549
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.194.keytab
--principal=oryx/127.9.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:44835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:20.851789 18006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:20.852104 18006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:20.852209 18006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:20.859922 18006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:20.860046 18006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:20.860132 18006 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:44:20.860222 18006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:44:20.870560 18006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:44835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.194.keytab
--principal=oryx/127.9.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.238.254:41549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:20.871901 18006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:20.873253 18006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:20.887629 18011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:20.887707 18014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:20.887730 18012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:20.889128 18006 server_base.cc:1034] running on GCE node
I20260926 16:44:20.889787 18006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:20.890875 18006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:20.892184 18006 hybrid_clock.cc:648] HybridClock initialized: now 1790441060892104 us; error 54 us; skew 500 ppm
I20260926 16:44:20.892519 18006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:20 dist-test-slave-wt2j krb5kdc[17750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441060, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:20.896008 18006 init.cc:377] Logged in from keytab as oryx/127.9.238.194@KRBTEST.COM (short username oryx)
I20260926 16:44:20.897724 18006 webserver.cc:460] Webserver started at http://127.9.238.194:43311/ using document root <none> and password file <none>
I20260926 16:44:20.898435 18006 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:20.898533 18006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:20.898808 18006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:20.901883 18006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-1/data/instance:
uuid: "53c8f79078f94da78570538801b43329"
format_stamp: "Formatted at 2026-09-26 16:44:20 on dist-test-slave-wt2j"
server_key: "df51ebede35ce45ee8917b6d0d21de6b"
server_key_iv: "9b37ad4ac1e6b488b41172b6a5247aae"
server_key_version: "encryptionkey@0"
I20260926 16:44:20.902647 18006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance:
uuid: "53c8f79078f94da78570538801b43329"
format_stamp: "Formatted at 2026-09-26 16:44:20 on dist-test-slave-wt2j"
server_key: "df51ebede35ce45ee8917b6d0d21de6b"
server_key_iv: "9b37ad4ac1e6b488b41172b6a5247aae"
server_key_version: "encryptionkey@0"
I20260926 16:44:20.908424 18006 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 16:44:20.912906 18021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:20.915318 18006 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s
I20260926 16:44:20.915512 18006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "53c8f79078f94da78570538801b43329"
format_stamp: "Formatted at 2026-09-26 16:44:20 on dist-test-slave-wt2j"
server_key: "df51ebede35ce45ee8917b6d0d21de6b"
server_key_iv: "9b37ad4ac1e6b488b41172b6a5247aae"
server_key_version: "encryptionkey@0"
I20260926 16:44:20.915851 18006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:20.944391 18006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:20.949450 18006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:20.949731 18006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:20.954172 18006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:20.954303 18006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:20.954558 18006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:20.954737 18006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:44:20 dist-test-slave-wt2j krb5kdc[17750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441060, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM
I20260926 16:44:20.987337 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:20.955250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:59205 (local address 127.9.238.254:41549)
0926 16:44:20.956302 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:20.956310 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:20.957968 (+ 1658us) server_negotiation.cc:407] Connection header received
0926 16:44:20.961532 (+ 3564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:20.961539 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:20.961674 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:20.961834 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:20.965328 (+ 3494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:20.966594 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:20.969800 (+ 3206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:20.970673 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:20.970918 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:20.978054 (+ 7136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:20.978065 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:20.978073 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:20.978132 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:20.982267 (+ 4135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:20.983371 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:20.983380 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:20.983385 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:20.983628 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:20.984628 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:20.984633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:20.984636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:20.985295 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:20.985526 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:20.986545 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0926 16:44:20.986873 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":349,"threads_started":1}
I20260926 16:44:20.989174 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:20.956396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:41549 (local address 127.9.238.194:59205)
0926 16:44:20.957655 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:20.957723 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:20.961028 (+ 3305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:20.962307 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:20.962324 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:20.963729 (+ 1405us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:20.965040 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:20.965078 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:20.966801 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:20.966808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:20.969473 (+ 2665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:20.969502 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:20.971087 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:20.971099 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:20.971494 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:20.972992 (+ 1498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:20.973053 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:20.977641 (+ 4588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:20.982491 (+ 4850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:20.982502 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:20.982523 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:20.983158 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:20.983849 (+ 691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:20.983859 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:20.983865 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:20.984411 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:20.985490 (+ 1079us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:20.985500 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:20.986069 (+ 569us) client_negotiation.cc:769] Sending connection context
0926 16:44:20.986432 (+ 363us) client_negotiation.cc:240] Negotiation successful
0926 16:44:20.987192 (+ 760us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"thread_start_us":492,"threads_started":1}
I20260926 16:44:21.010764 18006 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:34493
I20260926 16:44:21.010793 18141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:34493 every 8 connection(s)
I20260926 16:44:21.012931 18006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:44:21.016086 18142 heartbeater.cc:347] Connected to a master server at 127.9.238.254:41549
I20260926 16:44:21.016453 18142 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:21.017402 18030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:20.997374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:38055 (local address 127.9.238.254:41549)
0926 16:44:20.997670 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:20.997677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:20.998773 (+ 1096us) server_negotiation.cc:407] Connection header received
0926 16:44:20.999047 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:20.999057 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:20.999183 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:20.999370 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:21.001061 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:21.002481 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:21.005407 (+ 2926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:21.006307 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:21.006517 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:21.009257 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:21.009264 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:21.009269 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:21.009317 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:21.013547 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:21.014754 (+ 1207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:21.014764 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:21.014771 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:21.014975 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:21.015630 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:21.015637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:21.015640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:21.016067 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:21.016190 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:21.016854 (+ 664us) server_negotiation.cc:299] Negotiation successful
0926 16:44:21.017086 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260926 16:44:21.017454 18142 heartbeater.cc:511] Master 127.9.238.254:41549 requested a full tablet report, sending...
I20260926 16:44:21.017552 18114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:20.997531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:41549 (local address 127.9.238.194:38055)
0926 16:44:20.998605 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:20.998630 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:20.998813 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:20.999439 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:20.999445 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:21.000414 (+ 969us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:21.000836 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:21.000847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:21.002720 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:21.002728 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:21.005096 (+ 2368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:21.005122 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:21.006578 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:21.006588 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:21.006818 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:21.007948 (+ 1130us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:21.007981 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:21.008944 (+ 963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:21.013788 (+ 4844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:21.013795 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:21.013802 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:21.014549 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:21.015124 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:21.015132 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:21.015139 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:21.015395 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:21.016212 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:21.016218 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:21.016463 (+ 245us) client_negotiation.cc:769] Sending connection context
0926 16:44:21.016697 (+ 234us) client_negotiation.cc:240] Negotiation successful
0926 16:44:21.017053 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"thread_start_us":740,"threads_started":1}
I20260926 16:44:21.019140 17795 ts_manager.cc:194] Registered new tserver with Master: 53c8f79078f94da78570538801b43329 (127.9.238.194:34493)
I20260926 16:44:21.019481 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 18006
I20260926 16:44:21.019692 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance
I20260926 16:44:21.020255 10171 external_mini_cluster.cc:1428] Setting key f57bc1c7c976ce74c2bb5147270bf441
I20260926 16:44:21.020287 17795 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.238.194@KRBTEST.COM'} at 127.9.238.194:59205
WARNING: no policy specified for oryx/127.9.238.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.195.keytab.
Entry for principal oryx/127.9.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.195.keytab.
I20260926 16:44:21.080405 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:0
--local_ip_for_outbound_sockets=127.9.238.195
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:41549
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.195.keytab
--principal=oryx/127.9.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:44835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:44:21.435348 18002 heartbeater.cc:503] Master 127.9.238.254:41549 was elected leader, sending a full tablet report...
W20260926 16:44:21.501921 18146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:21.502291 18146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:21.502418 18146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:21.510910 18146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:21.511062 18146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:21.511153 18146 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 16:44:21.511240 18146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:44:21.522282 18146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:44835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.195.keytab
--principal=oryx/127.9.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.9.238.254:41549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:21.523823 18146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:21.525272 18146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:21.540747 18154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:21.540932 18152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:21.540792 18151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:21.542940 18146 server_base.cc:1034] running on GCE node
I20260926 16:44:21.543866 18146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:21.544974 18146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:21.546253 18146 hybrid_clock.cc:648] HybridClock initialized: now 1790441061546165 us; error 60 us; skew 500 ppm
I20260926 16:44:21.546710 18146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:21 dist-test-slave-wt2j krb5kdc[17750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441061, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:21.550501 18146 init.cc:377] Logged in from keytab as oryx/127.9.238.195@KRBTEST.COM (short username oryx)
I20260926 16:44:21.552578 18146 webserver.cc:460] Webserver started at http://127.9.238.195:35617/ using document root <none> and password file <none>
I20260926 16:44:21.553388 18146 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:21.553493 18146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:21.553869 18146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:21.557278 18146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-2/data/instance:
uuid: "858926b8a54f4578b2ed598e4e94343e"
format_stamp: "Formatted at 2026-09-26 16:44:21 on dist-test-slave-wt2j"
server_key: "53dbfaf566cd9d0d698212852eb92abc"
server_key_iv: "7ae33c78f3659894bb7fffcaff9684a2"
server_key_version: "encryptionkey@0"
I20260926 16:44:21.558236 18146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance:
uuid: "858926b8a54f4578b2ed598e4e94343e"
format_stamp: "Formatted at 2026-09-26 16:44:21 on dist-test-slave-wt2j"
server_key: "53dbfaf566cd9d0d698212852eb92abc"
server_key_iv: "7ae33c78f3659894bb7fffcaff9684a2"
server_key_version: "encryptionkey@0"
I20260926 16:44:21.564764 18146 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 16:44:21.569090 18161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:21.570981 18146 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 16:44:21.571213 18146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "858926b8a54f4578b2ed598e4e94343e"
format_stamp: "Formatted at 2026-09-26 16:44:21 on dist-test-slave-wt2j"
server_key: "53dbfaf566cd9d0d698212852eb92abc"
server_key_iv: "7ae33c78f3659894bb7fffcaff9684a2"
server_key_version: "encryptionkey@0"
I20260926 16:44:21.571467 18146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:21.602228 18146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:21.607216 18146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:21.607530 18146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:21.610637 18146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:21.610751 18146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:21.610926 18146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:21.611007 18146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:44:21 dist-test-slave-wt2j krb5kdc[17750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441061, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM
I20260926 16:44:21.650997 18170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:21.611991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:50699 (local address 127.9.238.254:41549)
0926 16:44:21.624139 (+ 12148us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:21.624147 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:21.624175 (+ 28us) server_negotiation.cc:407] Connection header received
0926 16:44:21.626038 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:21.626045 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:21.626174 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:21.626337 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:21.628925 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:21.629820 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:21.632729 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:21.633434 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:21.633648 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:21.641385 (+ 7737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:21.641397 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:21.641593 (+ 196us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:21.641670 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:21.645743 (+ 4073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:21.647382 (+ 1639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:21.647388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:21.647391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:21.647690 (+ 299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:21.648676 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:21.648680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:21.648682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:21.649061 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:21.649246 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:21.650209 (+ 963us) server_negotiation.cc:299] Negotiation successful
0926 16:44:21.650516 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11995,"thread_start_us":11853,"threads_started":1}
I20260926 16:44:21.653062 18171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:21.612852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:41549 (local address 127.9.238.195:50699)
0926 16:44:21.623014 (+ 10162us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:21.623088 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:21.625487 (+ 2399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:21.626674 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:21.626687 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:21.627445 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:21.628655 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:21.628691 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:21.630329 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:21.630335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:21.632398 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:21.632426 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:21.633917 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:21.633936 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:21.634504 (+ 568us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:21.635881 (+ 1377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:21.635921 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:21.640763 (+ 4842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:21.646018 (+ 5255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:21.646037 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:21.646069 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:21.647036 (+ 967us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:21.647866 (+ 830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:21.647876 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:21.647880 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:21.648463 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:21.649220 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:21.649232 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:21.649781 (+ 549us) client_negotiation.cc:769] Sending connection context
0926 16:44:21.650116 (+ 335us) client_negotiation.cc:240] Negotiation successful
0926 16:44:21.650833 (+ 717us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9727,"thread_start_us":9251,"threads_started":1}
I20260926 16:44:21.683221 18146 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:34973
I20260926 16:44:21.683241 18281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:34973 every 8 connection(s)
I20260926 16:44:21.685185 18146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:44:21.685559 18170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:21.664038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:35987 (local address 127.9.238.254:41549)
0926 16:44:21.664374 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:21.664385 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:21.665466 (+ 1081us) server_negotiation.cc:407] Connection header received
0926 16:44:21.665855 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:21.665864 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:21.665987 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:21.666167 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:21.668936 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:21.670500 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:21.672705 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:21.673677 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:21.673863 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:21.678525 (+ 4662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:21.678535 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:21.678541 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:21.678604 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:21.681826 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:21.683197 (+ 1371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:21.683206 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:21.683211 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:21.683378 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:21.683899 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:21.683903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:21.683905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:21.684237 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:21.684404 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:21.684928 (+ 524us) server_negotiation.cc:299] Negotiation successful
0926 16:44:21.685167 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 16:44:21.685787 18244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:21.664098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:41549 (local address 127.9.238.195:35987)
0926 16:44:21.665192 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:21.665218 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:21.665628 (+ 410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:21.666453 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:21.666462 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:21.667610 (+ 1148us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:21.668410 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:21.668573 (+ 163us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:21.670750 (+ 2177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:21.670756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:21.672443 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:21.672470 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:21.674500 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:21.674521 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:21.674801 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:21.676147 (+ 1346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:21.676180 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:21.678173 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:21.682032 (+ 3859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:21.682047 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:21.682055 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:21.683011 (+ 956us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:21.683581 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:21.683587 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:21.683592 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:21.683763 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:21.684383 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:21.684390 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:21.684639 (+ 249us) client_negotiation.cc:769] Sending connection context
0926 16:44:21.684816 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 16:44:21.685217 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"thread_start_us":755,"threads_started":1}
I20260926 16:44:21.688138 18282 heartbeater.cc:347] Connected to a master server at 127.9.238.254:41549
I20260926 16:44:21.688409 18282 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:21.689296 18282 heartbeater.cc:511] Master 127.9.238.254:41549 requested a full tablet report, sending...
I20260926 16:44:21.691159 17795 ts_manager.cc:194] Registered new tserver with Master: 858926b8a54f4578b2ed598e4e94343e (127.9.238.195:34973)
I20260926 16:44:21.692270 17795 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.9.238.195@KRBTEST.COM'} at 127.9.238.195:50699
I20260926 16:44:21.693053 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 18146
I20260926 16:44:21.693364 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance
I20260926 16:44:21.694031 10171 external_mini_cluster.cc:1428] Setting key 79f1d0df4ce7b72743a838af04930096
I20260926 16:44:21.697979 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 16:44:21 dist-test-slave-wt2j krb5kdc[17750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441061, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 16:44:21 dist-test-slave-wt2j krb5kdc[17750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:21 dist-test-slave-wt2j krb5kdc[17750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:22.023692 18142 heartbeater.cc:503] Master 127.9.238.254:41549 was elected leader, sending a full tablet report...
I20260926 16:44:22.695730 18282 heartbeater.cc:503] Master 127.9.238.254:41549 was elected leader, sending a full tablet report...
W20260926 16:44:22.917179 18290 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.158s user 0.003s sys 0.014s
W20260926 16:44:22.918736 18170 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:21.751473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56824 (local address 127.9.238.254:41549)
0926 16:44:21.751864 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:21.751872 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:21.752167 (+ 295us) server_negotiation.cc:407] Connection header received
0926 16:44:21.752457 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:21.752463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:21.752552 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:21.752690 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:21.754579 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:21.755684 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:21.757375 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:21.757964 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:21.758105 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:22.918391 (+1160286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56824: BlockingRecv error: recv got EOF from 127.0.0.1:56824 (error 108)
Metrics: {"server-negotiator.queue_time_us":176}
Sep 26 16:44:22 dist-test-slave-wt2j krb5kdc[17750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441061, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM
I20260926 16:44:22.941987 18170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:22.924331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56834 (local address 127.9.238.254:41549)
0926 16:44:22.924665 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:22.924673 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:22.924864 (+ 191us) server_negotiation.cc:407] Connection header received
0926 16:44:22.925053 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:22.925058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:22.925145 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:22.925270 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:22.926583 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:22.927481 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:22.928937 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:22.929520 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:22.929670 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:22.934173 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:22.934180 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:22.934184 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:22.934278 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:22.937854 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:22.938824 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:22.938829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:22.938832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:22.939028 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:22.939469 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:22.939473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:22.939475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:22.939898 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:22.940655 (+ 757us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:22.941283 (+ 628us) server_negotiation.cc:299] Negotiation successful
0926 16:44:22.941713 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260926 16:44:22.948036 17795 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:44:22.956465 17795 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:44:22.992862 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:22.975007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:34493 (local address 127.0.0.1:55574)
0926 16:44:22.980911 (+ 5904us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:22.980964 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:22.981275 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:22.982372 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:22.982381 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:22.982426 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:22.982987 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:22.983009 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:22.985993 (+ 2984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:22.985998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:22.988369 (+ 2371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:22.988385 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:22.991355 (+ 2970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:22.991365 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:22.991726 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:22.991820 (+ 94us) client_negotiation.cc:769] Sending connection context
0926 16:44:22.991962 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 16:44:22.992354 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5613,"mutex_wait_us":5,"thread_start_us":6617,"threads_started":1}
I20260926 16:44:22.994159 18303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:22.975307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55574 (local address 127.9.238.194:34493)
0926 16:44:22.980240 (+ 4933us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:22.980256 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:22.981114 (+ 858us) server_negotiation.cc:407] Connection header received
0926 16:44:22.981672 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:22.981684 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:22.982049 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:22.983424 (+ 1375us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:22.983758 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:22.985812 (+ 2054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:22.988624 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:22.991042 (+ 2418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:22.993359 (+ 2317us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:22.993623 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:22.993709 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 16:44:22.993882 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4663,"thread_start_us":4479,"threads_started":1}
I20260926 16:44:22.994756 18304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:22.975841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46688 (local address 127.9.238.193:41345)
0926 16:44:22.976880 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:22.976895 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:22.982033 (+ 5138us) server_negotiation.cc:407] Connection header received
0926 16:44:22.982347 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:22.982356 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:22.982594 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:22.982728 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:22.983466 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:22.985504 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:22.988680 (+ 3176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:22.991422 (+ 2742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:22.991762 (+ 340us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:22.992099 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:22.994105 (+ 2006us) server_negotiation.cc:299] Negotiation successful
0926 16:44:22.994362 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":413,"threads_started":1}
I20260926 16:44:22.995661 18301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:22.975190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:41345 (local address 127.0.0.1:46688)
0926 16:44:22.981916 (+ 6726us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:22.981934 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:22.982146 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:22.982955 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:22.982958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:22.982978 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:22.983315 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:22.983322 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:22.985649 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:22.985657 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:22.988369 (+ 2712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:22.988385 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:22.993594 (+ 5209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:22.993603 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:22.993828 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:22.993864 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 16:44:22.995028 (+ 1164us) client_negotiation.cc:240] Negotiation successful
0926 16:44:22.995217 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6608,"mutex_wait_us":84,"thread_start_us":5275,"threads_started":1}
I20260926 16:44:22.997903 18072 tablet_service.cc:1467] Processing CreateTablet for tablet 42727856a15543dc9858a767c93d0010 (DEFAULT_TABLE table=test-table [id=abe6f170867d4ca9b3301b20636d6784]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:22.999490 18072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42727856a15543dc9858a767c93d0010. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:23.000977 17932 tablet_service.cc:1467] Processing CreateTablet for tablet 42727856a15543dc9858a767c93d0010 (DEFAULT_TABLE table=test-table [id=abe6f170867d4ca9b3301b20636d6784]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:23.002269 18306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:22.983668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:34973 (local address 127.0.0.1:49910)
0926 16:44:22.992895 (+ 9227us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:22.992925 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:22.993087 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:22.994011 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:22.994017 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:22.994044 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:22.994423 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:22.994435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:22.996725 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:22.996733 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:22.998881 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:22.998901 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:23.001252 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:23.001259 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:23.001455 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:23.001499 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 16:44:23.001580 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 16:44:23.001755 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8979,"spinlock_wait_cycles":86912,"thread_start_us":8994,"threads_started":1}
I20260926 16:44:23.003024 17932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42727856a15543dc9858a767c93d0010. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:23.003099 18305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:22.982557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49910 (local address 127.9.238.195:34973)
0926 16:44:22.992637 (+ 10080us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:22.992648 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:22.993233 (+ 585us) server_negotiation.cc:407] Connection header received
0926 16:44:22.993387 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:22.993397 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:22.993753 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:22.994659 (+ 906us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:22.995012 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:22.996584 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:22.999131 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:23.000936 (+ 1805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:23.002171 (+ 1235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:23.002451 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:23.002543 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 16:44:23.002770 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9869,"thread_start_us":9340,"threads_started":1}
I20260926 16:44:23.007586 18212 tablet_service.cc:1467] Processing CreateTablet for tablet 42727856a15543dc9858a767c93d0010 (DEFAULT_TABLE table=test-table [id=abe6f170867d4ca9b3301b20636d6784]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:23.009801 18212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42727856a15543dc9858a767c93d0010. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:23.019435 18307 tablet_bootstrap.cc:492] T 42727856a15543dc9858a767c93d0010 P 53c8f79078f94da78570538801b43329: Bootstrap starting.
I20260926 16:44:23.022296 18308 tablet_bootstrap.cc:492] T 42727856a15543dc9858a767c93d0010 P 09bb826eee674015b88848b1c2e9c54d: Bootstrap starting.
I20260926 16:44:23.023748 18307 tablet_bootstrap.cc:654] T 42727856a15543dc9858a767c93d0010 P 53c8f79078f94da78570538801b43329: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:23.025074 18307 log.cc:824] T 42727856a15543dc9858a767c93d0010 P 53c8f79078f94da78570538801b43329: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:23.026225 18308 tablet_bootstrap.cc:654] T 42727856a15543dc9858a767c93d0010 P 09bb826eee674015b88848b1c2e9c54d: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:23.026623 18310 tablet_bootstrap.cc:492] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e: Bootstrap starting.
I20260926 16:44:23.027817 18308 log.cc:824] T 42727856a15543dc9858a767c93d0010 P 09bb826eee674015b88848b1c2e9c54d: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:23.030179 18307 tablet_bootstrap.cc:492] T 42727856a15543dc9858a767c93d0010 P 53c8f79078f94da78570538801b43329: No bootstrap required, opened a new log
I20260926 16:44:23.030500 18307 ts_tablet_manager.cc:1397] T 42727856a15543dc9858a767c93d0010 P 53c8f79078f94da78570538801b43329: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260926 16:44:23.032109 18310 tablet_bootstrap.cc:654] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:23.033392 18310 log.cc:824] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:23.033473 18308 tablet_bootstrap.cc:492] T 42727856a15543dc9858a767c93d0010 P 09bb826eee674015b88848b1c2e9c54d: No bootstrap required, opened a new log
I20260926 16:44:23.033824 18308 ts_tablet_manager.cc:1397] T 42727856a15543dc9858a767c93d0010 P 09bb826eee674015b88848b1c2e9c54d: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260926 16:44:23.037101 18310 tablet_bootstrap.cc:492] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e: No bootstrap required, opened a new log
I20260926 16:44:23.037415 18310 ts_tablet_manager.cc:1397] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260926 16:44:23.040890 18307 raft_consensus.cc:348] T 42727856a15543dc9858a767c93d0010 P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } } peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } }
I20260926 16:44:23.041420 18307 raft_consensus.cc:374] T 42727856a15543dc9858a767c93d0010 P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:23.041543 18307 raft_consensus.cc:729] T 42727856a15543dc9858a767c93d0010 P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c8f79078f94da78570538801b43329, State: Initialized, Role: FOLLOWER
I20260926 16:44:23.042407 18308 raft_consensus.cc:348] T 42727856a15543dc9858a767c93d0010 P 09bb826eee674015b88848b1c2e9c54d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } } peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } }
I20260926 16:44:23.042997 18308 raft_consensus.cc:374] T 42727856a15543dc9858a767c93d0010 P 09bb826eee674015b88848b1c2e9c54d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:23.043113 18308 raft_consensus.cc:729] T 42727856a15543dc9858a767c93d0010 P 09bb826eee674015b88848b1c2e9c54d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09bb826eee674015b88848b1c2e9c54d, State: Initialized, Role: FOLLOWER
I20260926 16:44:23.042425 18307 consensus_queue.cc:260] T 42727856a15543dc9858a767c93d0010 P 53c8f79078f94da78570538801b43329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } } peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } }
I20260926 16:44:23.043941 18308 consensus_queue.cc:260] T 42727856a15543dc9858a767c93d0010 P 09bb826eee674015b88848b1c2e9c54d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } } peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } }
I20260926 16:44:23.045616 18307 ts_tablet_manager.cc:1428] T 42727856a15543dc9858a767c93d0010 P 53c8f79078f94da78570538801b43329: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260926 16:44:23.047627 18308 ts_tablet_manager.cc:1428] T 42727856a15543dc9858a767c93d0010 P 09bb826eee674015b88848b1c2e9c54d: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260926 16:44:23.048134 18310 raft_consensus.cc:348] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } } peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } }
I20260926 16:44:23.048624 18310 raft_consensus.cc:374] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:23.048768 18310 raft_consensus.cc:729] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 858926b8a54f4578b2ed598e4e94343e, State: Initialized, Role: FOLLOWER
I20260926 16:44:23.049881 18310 consensus_queue.cc:260] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } } peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } }
I20260926 16:44:23.052175 18310 ts_tablet_manager.cc:1428] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
W20260926 16:44:23.177547 18003 tablet.cc:2369] T 42727856a15543dc9858a767c93d0010 P 09bb826eee674015b88848b1c2e9c54d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:44:23.189358 18283 tablet.cc:2369] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:44:23.267596 18143 tablet.cc:2369] T 42727856a15543dc9858a767c93d0010 P 53c8f79078f94da78570538801b43329: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:44:23.283262 18315 raft_consensus.cc:482] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:44:23.283671 18315 raft_consensus.cc:504] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } } peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } }
I20260926 16:44:23.285539 18315 leader_election.cc:288] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53c8f79078f94da78570538801b43329 (127.9.238.194:34493), 09bb826eee674015b88848b1c2e9c54d (127.9.238.193:41345)
I20260926 16:44:23.293875 18303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:23.286296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:59445 (local address 127.9.238.194:34493)
0926 16:44:23.286524 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:23.286530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:23.287030 (+ 500us) server_negotiation.cc:407] Connection header received
0926 16:44:23.287303 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:23.287312 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:23.287396 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:23.287510 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:23.288444 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:23.289230 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:23.291678 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:23.292577 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:23.292787 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:23.292878 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:23.293451 (+ 573us) server_negotiation.cc:299] Negotiation successful
0926 16:44:23.293594 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 16:44:23.293874 18304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:23.286116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:41707 (local address 127.9.238.193:41345)
0926 16:44:23.286791 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:23.286800 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:23.287371 (+ 571us) server_negotiation.cc:407] Connection header received
0926 16:44:23.287605 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:23.287611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:23.287703 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:23.287830 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:23.288611 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:23.289492 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:23.291618 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:23.292589 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:23.292811 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:23.292926 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:23.293392 (+ 466us) server_negotiation.cc:299] Negotiation successful
0926 16:44:23.293559 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511}
I20260926 16:44:23.294010 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:23.286228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:41345 (local address 127.9.238.195:41707)
0926 16:44:23.287184 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:23.287223 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:23.287390 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:23.287860 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:23.287869 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:23.287890 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:23.288426 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:23.288438 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:23.289693 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:23.289698 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:23.291391 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:23.291406 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:23.292865 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:23.292872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:23.293184 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:23.293236 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 16:44:23.293330 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 16:44:23.293500 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":48,"thread_start_us":577,"threads_started":1}
I20260926 16:44:23.294171 18316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:23.285861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:34493 (local address 127.9.238.195:59445)
0926 16:44:23.286870 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:23.286894 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:23.287126 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:23.287592 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:23.287597 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:23.287628 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:23.288276 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:23.288296 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:23.289384 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:23.289390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:23.291365 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:23.291382 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:23.292926 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:23.292933 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:23.293247 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:23.293276 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 16:44:23.293413 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 16:44:23.293616 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":62,"thread_start_us":579,"threads_started":1}
I20260926 16:44:23.295763 18092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42727856a15543dc9858a767c93d0010" candidate_uuid: "858926b8a54f4578b2ed598e4e94343e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c8f79078f94da78570538801b43329" is_pre_election: true
I20260926 16:44:23.295696 17957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42727856a15543dc9858a767c93d0010" candidate_uuid: "858926b8a54f4578b2ed598e4e94343e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09bb826eee674015b88848b1c2e9c54d" is_pre_election: true
I20260926 16:44:23.296315 18092 raft_consensus.cc:2454] T 42727856a15543dc9858a767c93d0010 P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 858926b8a54f4578b2ed598e4e94343e in term 0.
I20260926 16:44:23.296336 17957 raft_consensus.cc:2454] T 42727856a15543dc9858a767c93d0010 P 09bb826eee674015b88848b1c2e9c54d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 858926b8a54f4578b2ed598e4e94343e in term 0.
I20260926 16:44:23.297492 18164 leader_election.cc:302] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53c8f79078f94da78570538801b43329, 858926b8a54f4578b2ed598e4e94343e; no voters:
I20260926 16:44:23.298213 18315 raft_consensus.cc:2781] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:44:23.298403 18315 raft_consensus.cc:482] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:44:23.298530 18315 raft_consensus.cc:3037] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:23.301151 18315 raft_consensus.cc:504] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } } peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } }
I20260926 16:44:23.301877 18315 leader_election.cc:288] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e [CANDIDATE]: Term 1 election: Requested vote from peers 53c8f79078f94da78570538801b43329 (127.9.238.194:34493), 09bb826eee674015b88848b1c2e9c54d (127.9.238.193:41345)
I20260926 16:44:23.302654 18092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42727856a15543dc9858a767c93d0010" candidate_uuid: "858926b8a54f4578b2ed598e4e94343e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c8f79078f94da78570538801b43329"
I20260926 16:44:23.302898 18092 raft_consensus.cc:3037] T 42727856a15543dc9858a767c93d0010 P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:23.302796 17957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42727856a15543dc9858a767c93d0010" candidate_uuid: "858926b8a54f4578b2ed598e4e94343e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09bb826eee674015b88848b1c2e9c54d"
I20260926 16:44:23.303159 17957 raft_consensus.cc:3037] T 42727856a15543dc9858a767c93d0010 P 09bb826eee674015b88848b1c2e9c54d [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:23.305688 18092 raft_consensus.cc:2454] T 42727856a15543dc9858a767c93d0010 P 53c8f79078f94da78570538801b43329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 858926b8a54f4578b2ed598e4e94343e in term 1.
I20260926 16:44:23.306038 17957 raft_consensus.cc:2454] T 42727856a15543dc9858a767c93d0010 P 09bb826eee674015b88848b1c2e9c54d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 858926b8a54f4578b2ed598e4e94343e in term 1.
I20260926 16:44:23.306414 18164 leader_election.cc:302] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53c8f79078f94da78570538801b43329, 858926b8a54f4578b2ed598e4e94343e; no voters:
I20260926 16:44:23.306836 18315 raft_consensus.cc:2781] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:44:23.307426 18315 raft_consensus.cc:686] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e [term 1 LEADER]: Becoming Leader. State: Replica: 858926b8a54f4578b2ed598e4e94343e, State: Running, Role: LEADER
I20260926 16:44:23.308034 18315 consensus_queue.cc:237] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } } peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } }
I20260926 16:44:23.313707 17794 catalog_manager.cc:5495] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e reported cstate change: term changed from 0 to 1, leader changed from <none> to 858926b8a54f4578b2ed598e4e94343e (127.9.238.195). New cstate: current_term: 1 leader_uuid: "858926b8a54f4578b2ed598e4e94343e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } health_report { overall_health: UNKNOWN } } }
I20260926 16:44:23.364322 18305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:23.357429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49912 (local address 127.9.238.195:34973)
0926 16:44:23.357829 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:23.357837 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:23.357961 (+ 124us) server_negotiation.cc:407] Connection header received
0926 16:44:23.358257 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:23.358263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:23.358370 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:23.358550 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:44:23.359008 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:23.359933 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:23.361480 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:23.362010 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:23.362126 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:23.362740 (+ 614us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:44:23.363612 (+ 872us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:44:23.363740 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:23.363870 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 16:44:23.363958 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260926 16:44:23.374447 18092 raft_consensus.cc:1261] T 42727856a15543dc9858a767c93d0010 P 53c8f79078f94da78570538801b43329 [term 1 FOLLOWER]: Refusing update from remote peer 858926b8a54f4578b2ed598e4e94343e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:44:23.374447 17957 raft_consensus.cc:1261] T 42727856a15543dc9858a767c93d0010 P 09bb826eee674015b88848b1c2e9c54d [term 1 FOLLOWER]: Refusing update from remote peer 858926b8a54f4578b2ed598e4e94343e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:44:23.375725 18318 consensus_queue.cc:1035] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e [LEADER]: Connected to new peer: Peer: permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:44:23.376266 18315 consensus_queue.cc:1035] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e [LEADER]: Connected to new peer: Peer: permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:44:23.408830 17795 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:44:23.409202 17795 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:44:23.414345 17795 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fcfbc557bbb54548b890bd7cc48574b4: Sending DeleteTablet for 3 replicas of tablet 42727856a15543dc9858a767c93d0010
I20260926 16:44:23.416115 17932 tablet_service.cc:1514] Processing DeleteTablet for tablet 42727856a15543dc9858a767c93d0010 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:23 UTC) from {username='oryx', principal='oryx/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:46688
I20260926 16:44:23.416115 18072 tablet_service.cc:1514] Processing DeleteTablet for tablet 42727856a15543dc9858a767c93d0010 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:23 UTC) from {username='oryx', principal='oryx/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:55574
I20260926 16:44:23.416143 18212 tablet_service.cc:1514] Processing DeleteTablet for tablet 42727856a15543dc9858a767c93d0010 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:23 UTC) from {username='oryx', principal='oryx/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:49910
I20260926 16:44:23.417582 17795 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:44:23.419162 17795 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:44:23.428447 18333 tablet_replica.cc:330] T 42727856a15543dc9858a767c93d0010 P 09bb826eee674015b88848b1c2e9c54d: stopping tablet replica
I20260926 16:44:23.430014 18332 tablet_replica.cc:330] T 42727856a15543dc9858a767c93d0010 P 53c8f79078f94da78570538801b43329: stopping tablet replica
I20260926 16:44:23.430501 18332 raft_consensus.cc:2229] T 42727856a15543dc9858a767c93d0010 P 53c8f79078f94da78570538801b43329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:44:23.430987 18332 raft_consensus.cc:2258] T 42727856a15543dc9858a767c93d0010 P 53c8f79078f94da78570538801b43329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:44:23.431226 18333 raft_consensus.cc:2229] T 42727856a15543dc9858a767c93d0010 P 09bb826eee674015b88848b1c2e9c54d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:44:23.432039 18333 raft_consensus.cc:2258] T 42727856a15543dc9858a767c93d0010 P 09bb826eee674015b88848b1c2e9c54d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:44:23.433182 18331 tablet_replica.cc:330] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e: stopping tablet replica
I20260926 16:44:23.433856 18331 raft_consensus.cc:2229] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:44:23.435992 18333 ts_tablet_manager.cc:1905] T 42727856a15543dc9858a767c93d0010 P 09bb826eee674015b88848b1c2e9c54d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:44:23.437008 18331 raft_consensus.cc:2258] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:44:23.441823 18072 tablet_service.cc:1467] Processing CreateTablet for tablet 80236d3fbf4249d497b0005565d49ddf (DEFAULT_TABLE table=test-table [id=a039c22640d745d1a60eb8e2904a7b81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:23.442077 18212 tablet_service.cc:1467] Processing CreateTablet for tablet 80236d3fbf4249d497b0005565d49ddf (DEFAULT_TABLE table=test-table [id=a039c22640d745d1a60eb8e2904a7b81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:23.442338 17932 tablet_service.cc:1467] Processing CreateTablet for tablet 80236d3fbf4249d497b0005565d49ddf (DEFAULT_TABLE table=test-table [id=a039c22640d745d1a60eb8e2904a7b81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:23.442847 17932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80236d3fbf4249d497b0005565d49ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:23.442452 18072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80236d3fbf4249d497b0005565d49ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:23.443641 18332 ts_tablet_manager.cc:1905] T 42727856a15543dc9858a767c93d0010 P 53c8f79078f94da78570538801b43329: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:44:23.451002 18308 tablet_bootstrap.cc:492] T 80236d3fbf4249d497b0005565d49ddf P 09bb826eee674015b88848b1c2e9c54d: Bootstrap starting.
I20260926 16:44:23.451769 18212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80236d3fbf4249d497b0005565d49ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:23.453100 18332 ts_tablet_manager.cc:1918] T 42727856a15543dc9858a767c93d0010 P 53c8f79078f94da78570538801b43329: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:44:23.453191 18308 tablet_bootstrap.cc:654] T 80236d3fbf4249d497b0005565d49ddf P 09bb826eee674015b88848b1c2e9c54d: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:23.457252 18331 ts_tablet_manager.cc:1905] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:44:23.461328 18310 tablet_bootstrap.cc:492] T 80236d3fbf4249d497b0005565d49ddf P 858926b8a54f4578b2ed598e4e94343e: Bootstrap starting.
I20260926 16:44:23.462025 18333 ts_tablet_manager.cc:1918] T 42727856a15543dc9858a767c93d0010 P 09bb826eee674015b88848b1c2e9c54d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:44:23.462327 18333 log.cc:1196] T 42727856a15543dc9858a767c93d0010 P 09bb826eee674015b88848b1c2e9c54d: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-0/wal/wals/42727856a15543dc9858a767c93d0010
I20260926 16:44:23.463282 18333 ts_tablet_manager.cc:1939] T 42727856a15543dc9858a767c93d0010 P 09bb826eee674015b88848b1c2e9c54d: Deleting consensus metadata
I20260926 16:44:23.463378 18310 tablet_bootstrap.cc:654] T 80236d3fbf4249d497b0005565d49ddf P 858926b8a54f4578b2ed598e4e94343e: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:23.464807 18308 tablet_bootstrap.cc:492] T 80236d3fbf4249d497b0005565d49ddf P 09bb826eee674015b88848b1c2e9c54d: No bootstrap required, opened a new log
I20260926 16:44:23.465011 18308 ts_tablet_manager.cc:1397] T 80236d3fbf4249d497b0005565d49ddf P 09bb826eee674015b88848b1c2e9c54d: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260926 16:44:23.465951 17780 catalog_manager.cc:4841] TS 09bb826eee674015b88848b1c2e9c54d (127.9.238.193:41345): tablet 42727856a15543dc9858a767c93d0010 (table test-table [id=abe6f170867d4ca9b3301b20636d6784]) successfully deleted
I20260926 16:44:23.466086 18308 raft_consensus.cc:348] T 80236d3fbf4249d497b0005565d49ddf P 09bb826eee674015b88848b1c2e9c54d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:23.466361 18308 raft_consensus.cc:374] T 80236d3fbf4249d497b0005565d49ddf P 09bb826eee674015b88848b1c2e9c54d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:23.466504 18308 raft_consensus.cc:729] T 80236d3fbf4249d497b0005565d49ddf P 09bb826eee674015b88848b1c2e9c54d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09bb826eee674015b88848b1c2e9c54d, State: Initialized, Role: FOLLOWER
I20260926 16:44:23.455787 18307 tablet_bootstrap.cc:492] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329: Bootstrap starting.
I20260926 16:44:23.467100 18308 consensus_queue.cc:260] T 80236d3fbf4249d497b0005565d49ddf P 09bb826eee674015b88848b1c2e9c54d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:23.468391 18310 tablet_bootstrap.cc:492] T 80236d3fbf4249d497b0005565d49ddf P 858926b8a54f4578b2ed598e4e94343e: No bootstrap required, opened a new log
I20260926 16:44:23.468585 18310 ts_tablet_manager.cc:1397] T 80236d3fbf4249d497b0005565d49ddf P 858926b8a54f4578b2ed598e4e94343e: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260926 16:44:23.469647 18307 tablet_bootstrap.cc:654] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:23.469544 18310 raft_consensus.cc:348] T 80236d3fbf4249d497b0005565d49ddf P 858926b8a54f4578b2ed598e4e94343e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:23.469816 18310 raft_consensus.cc:374] T 80236d3fbf4249d497b0005565d49ddf P 858926b8a54f4578b2ed598e4e94343e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:23.469920 18310 raft_consensus.cc:729] T 80236d3fbf4249d497b0005565d49ddf P 858926b8a54f4578b2ed598e4e94343e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 858926b8a54f4578b2ed598e4e94343e, State: Initialized, Role: FOLLOWER
I20260926 16:44:23.470238 18310 consensus_queue.cc:260] T 80236d3fbf4249d497b0005565d49ddf P 858926b8a54f4578b2ed598e4e94343e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:23.472460 18310 ts_tablet_manager.cc:1428] T 80236d3fbf4249d497b0005565d49ddf P 858926b8a54f4578b2ed598e4e94343e: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260926 16:44:23.473263 18332 log.cc:1196] T 42727856a15543dc9858a767c93d0010 P 53c8f79078f94da78570538801b43329: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-1/wal/wals/42727856a15543dc9858a767c93d0010
I20260926 16:44:23.474125 18332 ts_tablet_manager.cc:1939] T 42727856a15543dc9858a767c93d0010 P 53c8f79078f94da78570538801b43329: Deleting consensus metadata
I20260926 16:44:23.474704 18307 tablet_bootstrap.cc:492] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329: No bootstrap required, opened a new log
I20260926 16:44:23.474956 18307 ts_tablet_manager.cc:1397] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.000s
I20260926 16:44:23.476150 18307 raft_consensus.cc:348] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:23.476503 18307 raft_consensus.cc:374] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:23.476676 18307 raft_consensus.cc:729] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c8f79078f94da78570538801b43329, State: Initialized, Role: FOLLOWER
I20260926 16:44:23.477036 17781 catalog_manager.cc:4841] TS 53c8f79078f94da78570538801b43329 (127.9.238.194:34493): tablet 42727856a15543dc9858a767c93d0010 (table test-table [id=abe6f170867d4ca9b3301b20636d6784]) successfully deleted
I20260926 16:44:23.477236 18307 consensus_queue.cc:260] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:23.478206 18308 ts_tablet_manager.cc:1428] T 80236d3fbf4249d497b0005565d49ddf P 09bb826eee674015b88848b1c2e9c54d: Time spent starting tablet: real 0.013s user 0.002s sys 0.001s
I20260926 16:44:23.478591 18307 ts_tablet_manager.cc:1428] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 16:44:23.484098 18331 ts_tablet_manager.cc:1918] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:44:23.484325 18331 log.cc:1196] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-2/wal/wals/42727856a15543dc9858a767c93d0010
I20260926 16:44:23.484966 18331 ts_tablet_manager.cc:1939] T 42727856a15543dc9858a767c93d0010 P 858926b8a54f4578b2ed598e4e94343e: Deleting consensus metadata
I20260926 16:44:23.487094 17781 catalog_manager.cc:4841] TS 858926b8a54f4578b2ed598e4e94343e (127.9.238.195:34973): tablet 42727856a15543dc9858a767c93d0010 (table test-table [id=abe6f170867d4ca9b3301b20636d6784]) successfully deleted
I20260926 16:44:23.646189 18313 raft_consensus.cc:482] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:44:23.646498 18313 raft_consensus.cc:504] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:23.648842 18313 leader_election.cc:288] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09bb826eee674015b88848b1c2e9c54d (127.9.238.193:41345), 858926b8a54f4578b2ed598e4e94343e (127.9.238.195:34973)
I20260926 16:44:23.659073 18304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:23.649658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:38113 (local address 127.9.238.193:41345)
0926 16:44:23.649895 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:23.649903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:23.650030 (+ 127us) server_negotiation.cc:407] Connection header received
0926 16:44:23.650408 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:23.650416 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:23.650508 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:23.650587 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:23.651987 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:23.653055 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:23.656336 (+ 3281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:23.657817 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:23.658034 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:23.658122 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:23.658515 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 16:44:23.658698 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260926 16:44:23.659181 18337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:23.649178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:41345 (local address 127.9.238.194:38113)
0926 16:44:23.649894 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:23.649919 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:23.650179 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:23.651107 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:23.651113 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:23.651151 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:23.651740 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:23.651753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:23.653207 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:23.653214 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:23.656101 (+ 2887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:23.656119 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:23.658121 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:23.658129 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:23.658311 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:23.658358 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 16:44:23.658476 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 16:44:23.658710 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":25,"spinlock_wait_cycles":34688,"thread_start_us":1205,"threads_started":1}
I20260926 16:44:23.659569 18305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:23.650308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:54259 (local address 127.9.238.195:34973)
0926 16:44:23.650956 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:23.650966 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:23.651319 (+ 353us) server_negotiation.cc:407] Connection header received
0926 16:44:23.651537 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:23.651575 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:23.651702 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:23.651831 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:23.652502 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:23.653654 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:23.656672 (+ 3018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:23.657913 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:23.658136 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:23.658305 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:23.658924 (+ 619us) server_negotiation.cc:299] Negotiation successful
0926 16:44:23.659139 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469}
I20260926 16:44:23.660437 18338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:23.649307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:34973 (local address 127.9.238.194:54259)
0926 16:44:23.651156 (+ 1849us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:23.651177 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:23.651359 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:23.651865 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:23.651872 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:23.651901 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:23.652307 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:23.652320 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:23.653852 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:23.653860 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:23.656450 (+ 2590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:23.656459 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:23.658399 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:23.658408 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:23.658671 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:23.658718 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 16:44:23.659813 (+ 1095us) client_negotiation.cc:240] Negotiation successful
0926 16:44:23.660007 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1713,"mutex_wait_us":46,"spinlock_wait_cycles":768,"thread_start_us":1441,"threads_started":1}
I20260926 16:44:23.660485 17957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80236d3fbf4249d497b0005565d49ddf" candidate_uuid: "53c8f79078f94da78570538801b43329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09bb826eee674015b88848b1c2e9c54d" is_pre_election: true
I20260926 16:44:23.660856 17957 raft_consensus.cc:2454] T 80236d3fbf4249d497b0005565d49ddf P 09bb826eee674015b88848b1c2e9c54d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53c8f79078f94da78570538801b43329 in term 0.
I20260926 16:44:23.661491 18023 leader_election.cc:302] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09bb826eee674015b88848b1c2e9c54d, 53c8f79078f94da78570538801b43329; no voters:
I20260926 16:44:23.661603 18236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80236d3fbf4249d497b0005565d49ddf" candidate_uuid: "53c8f79078f94da78570538801b43329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "858926b8a54f4578b2ed598e4e94343e" is_pre_election: true
I20260926 16:44:23.662113 18313 raft_consensus.cc:2781] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:44:23.662263 18313 raft_consensus.cc:482] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:44:23.662189 18236 raft_consensus.cc:2454] T 80236d3fbf4249d497b0005565d49ddf P 858926b8a54f4578b2ed598e4e94343e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53c8f79078f94da78570538801b43329 in term 0.
I20260926 16:44:23.662393 18313 raft_consensus.cc:3037] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:23.665757 18313 raft_consensus.cc:504] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:23.666610 18313 leader_election.cc:288] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329 [CANDIDATE]: Term 1 election: Requested vote from peers 09bb826eee674015b88848b1c2e9c54d (127.9.238.193:41345), 858926b8a54f4578b2ed598e4e94343e (127.9.238.195:34973)
I20260926 16:44:23.667402 17957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80236d3fbf4249d497b0005565d49ddf" candidate_uuid: "53c8f79078f94da78570538801b43329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09bb826eee674015b88848b1c2e9c54d"
I20260926 16:44:23.667464 18236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80236d3fbf4249d497b0005565d49ddf" candidate_uuid: "53c8f79078f94da78570538801b43329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "858926b8a54f4578b2ed598e4e94343e"
I20260926 16:44:23.667754 17957 raft_consensus.cc:3037] T 80236d3fbf4249d497b0005565d49ddf P 09bb826eee674015b88848b1c2e9c54d [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:23.667874 18236 raft_consensus.cc:3037] T 80236d3fbf4249d497b0005565d49ddf P 858926b8a54f4578b2ed598e4e94343e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:23.670941 18236 raft_consensus.cc:2454] T 80236d3fbf4249d497b0005565d49ddf P 858926b8a54f4578b2ed598e4e94343e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53c8f79078f94da78570538801b43329 in term 1.
I20260926 16:44:23.671090 17957 raft_consensus.cc:2454] T 80236d3fbf4249d497b0005565d49ddf P 09bb826eee674015b88848b1c2e9c54d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53c8f79078f94da78570538801b43329 in term 1.
I20260926 16:44:23.671800 18024 leader_election.cc:302] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 09bb826eee674015b88848b1c2e9c54d, 53c8f79078f94da78570538801b43329, 858926b8a54f4578b2ed598e4e94343e; no voters:
I20260926 16:44:23.672289 18313 raft_consensus.cc:2781] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:44:23.672901 18313 raft_consensus.cc:686] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329 [term 1 LEADER]: Becoming Leader. State: Replica: 53c8f79078f94da78570538801b43329, State: Running, Role: LEADER
I20260926 16:44:23.673393 18313 consensus_queue.cc:237] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:23.678442 17793 catalog_manager.cc:5495] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53c8f79078f94da78570538801b43329 (127.9.238.194). New cstate: current_term: 1 leader_uuid: "53c8f79078f94da78570538801b43329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } health_report { overall_health: UNKNOWN } } }
Sep 26 16:44:23 dist-test-slave-wt2j krb5kdc[17750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441059, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.254@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM
I20260926 16:44:23.747990 18347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:23.730895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56848 (local address 127.9.238.254:41549)
0926 16:44:23.732109 (+ 1214us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:23.732118 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:23.732145 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:44:23.732206 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:23.732210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:23.732273 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:23.732401 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:23.733762 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:23.734623 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:23.736001 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:23.736446 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:23.736590 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:23.741849 (+ 5259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:23.741854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:23.741858 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:23.741890 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:23.744981 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:23.745766 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:23.745771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:23.745780 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:23.745891 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:23.746253 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:23.746257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:23.746260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:23.746568 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:23.746745 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:23.747239 (+ 494us) server_negotiation.cc:299] Negotiation successful
0926 16:44:23.747488 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1070,"thread_start_us":844,"threads_started":1}
I20260926 16:44:23.748107 18346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:23.730814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:41549 (local address 127.0.0.1:56848)
0926 16:44:23.731727 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:23.731765 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:23.732042 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:23.732432 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:23.732436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:23.733321 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:23.733557 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:23.733566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:23.734780 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:23.734786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:23.735867 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:23.735879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:23.736616 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:23.736621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:23.736823 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:23.737887 (+ 1064us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:23.737909 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:23.741593 (+ 3684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:23.745139 (+ 3546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:23.745160 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:23.745170 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:23.745640 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:23.745989 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:23.745996 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:23.746001 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:23.746146 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:23.746702 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:23.746712 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:23.746947 (+ 235us) client_negotiation.cc:769] Sending connection context
0926 16:44:23.747164 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 16:44:23.747686 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":133,"thread_start_us":483,"threads_started":1}
I20260926 16:44:23.754660 17793 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:56848:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 16:44:23.756253 17793 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:44:23.765908 17932 tablet_service.cc:1467] Processing CreateTablet for tablet 588a1afc7f0d4705b88f3c7bae9032ca (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=69b131aad08547a1a9302c222707c600]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:44:23.766667 17932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 588a1afc7f0d4705b88f3c7bae9032ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:23.766573 18212 tablet_service.cc:1467] Processing CreateTablet for tablet 588a1afc7f0d4705b88f3c7bae9032ca (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=69b131aad08547a1a9302c222707c600]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:44:23.766588 18072 tablet_service.cc:1467] Processing CreateTablet for tablet 588a1afc7f0d4705b88f3c7bae9032ca (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=69b131aad08547a1a9302c222707c600]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 16:44:23.767241 18072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 588a1afc7f0d4705b88f3c7bae9032ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:23.767324 18212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 588a1afc7f0d4705b88f3c7bae9032ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:23.775568 18307 tablet_bootstrap.cc:492] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329: Bootstrap starting.
I20260926 16:44:23.775728 18308 tablet_bootstrap.cc:492] T 588a1afc7f0d4705b88f3c7bae9032ca P 09bb826eee674015b88848b1c2e9c54d: Bootstrap starting.
I20260926 16:44:23.777562 18307 tablet_bootstrap.cc:654] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:23.778074 18308 tablet_bootstrap.cc:654] T 588a1afc7f0d4705b88f3c7bae9032ca P 09bb826eee674015b88848b1c2e9c54d: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:23.779261 18310 tablet_bootstrap.cc:492] T 588a1afc7f0d4705b88f3c7bae9032ca P 858926b8a54f4578b2ed598e4e94343e: Bootstrap starting.
I20260926 16:44:23.781177 18310 tablet_bootstrap.cc:654] T 588a1afc7f0d4705b88f3c7bae9032ca P 858926b8a54f4578b2ed598e4e94343e: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:23.782248 18308 tablet_bootstrap.cc:492] T 588a1afc7f0d4705b88f3c7bae9032ca P 09bb826eee674015b88848b1c2e9c54d: No bootstrap required, opened a new log
I20260926 16:44:23.782471 18308 ts_tablet_manager.cc:1397] T 588a1afc7f0d4705b88f3c7bae9032ca P 09bb826eee674015b88848b1c2e9c54d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 16:44:23.782965 18307 tablet_bootstrap.cc:492] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329: No bootstrap required, opened a new log
I20260926 16:44:23.783098 18307 ts_tablet_manager.cc:1397] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260926 16:44:23.783403 18308 raft_consensus.cc:348] T 588a1afc7f0d4705b88f3c7bae9032ca P 09bb826eee674015b88848b1c2e9c54d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:23.783717 18308 raft_consensus.cc:374] T 588a1afc7f0d4705b88f3c7bae9032ca P 09bb826eee674015b88848b1c2e9c54d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:23.783805 18308 raft_consensus.cc:729] T 588a1afc7f0d4705b88f3c7bae9032ca P 09bb826eee674015b88848b1c2e9c54d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09bb826eee674015b88848b1c2e9c54d, State: Initialized, Role: FOLLOWER
I20260926 16:44:23.783720 18307 raft_consensus.cc:348] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:23.783913 18307 raft_consensus.cc:374] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:23.783977 18307 raft_consensus.cc:729] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c8f79078f94da78570538801b43329, State: Initialized, Role: FOLLOWER
I20260926 16:44:23.784147 18308 consensus_queue.cc:260] T 588a1afc7f0d4705b88f3c7bae9032ca P 09bb826eee674015b88848b1c2e9c54d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:23.784343 18307 consensus_queue.cc:260] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:23.784677 18314 tablet_replica.cc:439] T 588a1afc7f0d4705b88f3c7bae9032ca P 09bb826eee674015b88848b1c2e9c54d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } } }
I20260926 16:44:23.784808 18313 tablet_replica.cc:439] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } } }
I20260926 16:44:23.785074 18314 tablet_replica.cc:442] T 588a1afc7f0d4705b88f3c7bae9032ca P 09bb826eee674015b88848b1c2e9c54d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:44:23.785127 18313 tablet_replica.cc:442] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:44:23.786535 18307 ts_tablet_manager.cc:1428] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 16:44:23.786556 18308 ts_tablet_manager.cc:1428] T 588a1afc7f0d4705b88f3c7bae9032ca P 09bb826eee674015b88848b1c2e9c54d: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 16:44:23.788597 18310 tablet_bootstrap.cc:492] T 588a1afc7f0d4705b88f3c7bae9032ca P 858926b8a54f4578b2ed598e4e94343e: No bootstrap required, opened a new log
I20260926 16:44:23.788810 18310 ts_tablet_manager.cc:1397] T 588a1afc7f0d4705b88f3c7bae9032ca P 858926b8a54f4578b2ed598e4e94343e: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20260926 16:44:23.789470 18310 raft_consensus.cc:348] T 588a1afc7f0d4705b88f3c7bae9032ca P 858926b8a54f4578b2ed598e4e94343e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:23.789716 18310 raft_consensus.cc:374] T 588a1afc7f0d4705b88f3c7bae9032ca P 858926b8a54f4578b2ed598e4e94343e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:23.789835 18310 raft_consensus.cc:729] T 588a1afc7f0d4705b88f3c7bae9032ca P 858926b8a54f4578b2ed598e4e94343e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 858926b8a54f4578b2ed598e4e94343e, State: Initialized, Role: FOLLOWER
I20260926 16:44:23.790148 18310 consensus_queue.cc:260] T 588a1afc7f0d4705b88f3c7bae9032ca P 858926b8a54f4578b2ed598e4e94343e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:23.790642 18315 tablet_replica.cc:439] T 588a1afc7f0d4705b88f3c7bae9032ca P 858926b8a54f4578b2ed598e4e94343e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } } }
I20260926 16:44:23.790854 18310 ts_tablet_manager.cc:1428] T 588a1afc7f0d4705b88f3c7bae9032ca P 858926b8a54f4578b2ed598e4e94343e: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260926 16:44:23.790999 18315 tablet_replica.cc:442] T 588a1afc7f0d4705b88f3c7bae9032ca P 858926b8a54f4578b2ed598e4e94343e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:44:23.963045 18313 raft_consensus.cc:482] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:44:23.963363 18313 raft_consensus.cc:504] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:23.964143 18313 leader_election.cc:288] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09bb826eee674015b88848b1c2e9c54d (127.9.238.193:41345), 858926b8a54f4578b2ed598e4e94343e (127.9.238.195:34973)
I20260926 16:44:23.964855 17957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588a1afc7f0d4705b88f3c7bae9032ca" candidate_uuid: "53c8f79078f94da78570538801b43329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09bb826eee674015b88848b1c2e9c54d" is_pre_election: true
I20260926 16:44:23.964951 18236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588a1afc7f0d4705b88f3c7bae9032ca" candidate_uuid: "53c8f79078f94da78570538801b43329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "858926b8a54f4578b2ed598e4e94343e" is_pre_election: true
I20260926 16:44:23.965242 18236 raft_consensus.cc:2454] T 588a1afc7f0d4705b88f3c7bae9032ca P 858926b8a54f4578b2ed598e4e94343e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53c8f79078f94da78570538801b43329 in term 0.
I20260926 16:44:23.965241 17957 raft_consensus.cc:2454] T 588a1afc7f0d4705b88f3c7bae9032ca P 09bb826eee674015b88848b1c2e9c54d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53c8f79078f94da78570538801b43329 in term 0.
I20260926 16:44:23.965698 18024 leader_election.cc:302] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53c8f79078f94da78570538801b43329, 858926b8a54f4578b2ed598e4e94343e; no voters:
I20260926 16:44:23.965973 18313 raft_consensus.cc:2781] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:44:23.966076 18313 raft_consensus.cc:482] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:44:23.966167 18313 raft_consensus.cc:3037] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:23.968367 18313 raft_consensus.cc:504] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:23.969004 18313 leader_election.cc:288] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329 [CANDIDATE]: Term 1 election: Requested vote from peers 09bb826eee674015b88848b1c2e9c54d (127.9.238.193:41345), 858926b8a54f4578b2ed598e4e94343e (127.9.238.195:34973)
I20260926 16:44:23.969612 18236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588a1afc7f0d4705b88f3c7bae9032ca" candidate_uuid: "53c8f79078f94da78570538801b43329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "858926b8a54f4578b2ed598e4e94343e"
I20260926 16:44:23.969722 17957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "588a1afc7f0d4705b88f3c7bae9032ca" candidate_uuid: "53c8f79078f94da78570538801b43329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09bb826eee674015b88848b1c2e9c54d"
I20260926 16:44:23.969942 18236 raft_consensus.cc:3037] T 588a1afc7f0d4705b88f3c7bae9032ca P 858926b8a54f4578b2ed598e4e94343e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:23.970016 17957 raft_consensus.cc:3037] T 588a1afc7f0d4705b88f3c7bae9032ca P 09bb826eee674015b88848b1c2e9c54d [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:23.972235 17957 raft_consensus.cc:2454] T 588a1afc7f0d4705b88f3c7bae9032ca P 09bb826eee674015b88848b1c2e9c54d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53c8f79078f94da78570538801b43329 in term 1.
I20260926 16:44:23.972608 18236 raft_consensus.cc:2454] T 588a1afc7f0d4705b88f3c7bae9032ca P 858926b8a54f4578b2ed598e4e94343e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53c8f79078f94da78570538801b43329 in term 1.
I20260926 16:44:23.972929 18023 leader_election.cc:302] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09bb826eee674015b88848b1c2e9c54d, 53c8f79078f94da78570538801b43329; no voters:
I20260926 16:44:23.973297 18313 raft_consensus.cc:2781] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:44:23.973541 18313 raft_consensus.cc:686] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329 [term 1 LEADER]: Becoming Leader. State: Replica: 53c8f79078f94da78570538801b43329, State: Running, Role: LEADER
I20260926 16:44:23.973842 18313 consensus_queue.cc:237] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:23.975692 18339 tablet_replica.cc:439] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329: TxnStatusTablet state changed. Reason: New leader 53c8f79078f94da78570538801b43329. Latest consensus state: current_term: 1 leader_uuid: "53c8f79078f94da78570538801b43329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } } }
I20260926 16:44:23.976070 18339 tablet_replica.cc:442] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:44:23.976783 18352 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 16:44:23.977003 18352 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 16:44:23.977438 17794 catalog_manager.cc:5495] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53c8f79078f94da78570538801b43329 (127.9.238.194). New cstate: current_term: 1 leader_uuid: "53c8f79078f94da78570538801b43329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } health_report { overall_health: UNKNOWN } } }
I20260926 16:44:24.003814 18346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:23.993986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:34493 (local address 127.0.0.1:55586)
0926 16:44:23.994273 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:23.994428 (+ 155us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:23.994677 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:23.995374 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:23.995379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:23.995400 (+ 21us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:44:23.995701 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:23.995711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:23.997119 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:23.997128 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:23.998359 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:23.998373 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:23.999822 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:23.999828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:24.000157 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:24.000287 (+ 130us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:44:24.002114 (+ 1827us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:44:24.002144 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 16:44:24.002755 (+ 611us) client_negotiation.cc:240] Negotiation successful
0926 16:44:24.003104 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 16:44:24.004904 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:23.993937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.9.238.194:34493)
0926 16:44:23.994932 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:23.994937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:23.994958 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:44:23.995059 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:23.995063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:23.995174 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:23.995353 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:44:23.995888 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:23.996966 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:23.998540 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:23.999506 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:23.999732 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:24.000617 (+ 885us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:44:24.001936 (+ 1319us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:44:24.002495 (+ 559us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:24.004183 (+ 1688us) server_negotiation.cc:299] Negotiation successful
0926 16:44:24.004313 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":581,"threads_started":1}
I20260926 16:44:24.011394 17957 raft_consensus.cc:1261] T 588a1afc7f0d4705b88f3c7bae9032ca P 09bb826eee674015b88848b1c2e9c54d [term 1 FOLLOWER]: Refusing update from remote peer 53c8f79078f94da78570538801b43329: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:44:24.012010 18236 raft_consensus.cc:1261] T 588a1afc7f0d4705b88f3c7bae9032ca P 858926b8a54f4578b2ed598e4e94343e [term 1 FOLLOWER]: Refusing update from remote peer 53c8f79078f94da78570538801b43329: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:44:24.012580 18339 consensus_queue.cc:1035] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:44:24.013177 18313 consensus_queue.cc:1035] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:44:24.020969 18315 tablet_replica.cc:439] T 588a1afc7f0d4705b88f3c7bae9032ca P 858926b8a54f4578b2ed598e4e94343e: TxnStatusTablet state changed. Reason: New leader 53c8f79078f94da78570538801b43329. Latest consensus state: current_term: 1 leader_uuid: "53c8f79078f94da78570538801b43329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } } }
I20260926 16:44:24.021473 18315 tablet_replica.cc:442] T 588a1afc7f0d4705b88f3c7bae9032ca P 858926b8a54f4578b2ed598e4e94343e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:44:24.022800 18356 mvcc.cc:205] Tried to move back new op lower bound from 7333646598181736448 to 7333646598039474176. Current Snapshot: MvccSnapshot[applied={T|T < 7333646598181736448}]
I20260926 16:44:24.023918 18313 tablet_replica.cc:439] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "53c8f79078f94da78570538801b43329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } } }
I20260926 16:44:24.024315 18313 tablet_replica.cc:442] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:44:24.037236 18314 tablet_replica.cc:439] T 588a1afc7f0d4705b88f3c7bae9032ca P 09bb826eee674015b88848b1c2e9c54d: TxnStatusTablet state changed. Reason: New leader 53c8f79078f94da78570538801b43329. Latest consensus state: current_term: 1 leader_uuid: "53c8f79078f94da78570538801b43329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } } }
I20260926 16:44:24.037542 18314 tablet_replica.cc:442] T 588a1afc7f0d4705b88f3c7bae9032ca P 09bb826eee674015b88848b1c2e9c54d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:44:24.041245 18315 tablet_replica.cc:439] T 588a1afc7f0d4705b88f3c7bae9032ca P 858926b8a54f4578b2ed598e4e94343e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "53c8f79078f94da78570538801b43329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } } }
I20260926 16:44:24.041519 18315 tablet_replica.cc:442] T 588a1afc7f0d4705b88f3c7bae9032ca P 858926b8a54f4578b2ed598e4e94343e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:44:24.041303 18339 tablet_replica.cc:439] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "53c8f79078f94da78570538801b43329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } } }
I20260926 16:44:24.041651 18339 tablet_replica.cc:442] T 588a1afc7f0d4705b88f3c7bae9032ca P 53c8f79078f94da78570538801b43329: This TxnStatusTablet replica's current role is: LEADER
I20260926 16:44:24.048249 18314 tablet_replica.cc:439] T 588a1afc7f0d4705b88f3c7bae9032ca P 09bb826eee674015b88848b1c2e9c54d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "53c8f79078f94da78570538801b43329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } } }
I20260926 16:44:24.048581 18314 tablet_replica.cc:442] T 588a1afc7f0d4705b88f3c7bae9032ca P 09bb826eee674015b88848b1c2e9c54d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 16:44:24.057490 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:24.049469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55602 (local address 127.9.238.194:34493)
0926 16:44:24.049962 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:24.049972 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:24.049989 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:44:24.050140 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:24.050145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:24.050287 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:24.050423 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:44:24.051435 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:24.052921 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:24.054944 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:24.055725 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:24.055888 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:24.056366 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:44:24.056565 (+ 199us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:44:24.056664 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:24.056853 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 16:44:24.056954 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216}
I20260926 16:44:24.080073 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:24.072861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:36643 (local address 127.9.238.194:34493)
0926 16:44:24.073097 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:24.073102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:24.074002 (+ 900us) server_negotiation.cc:407] Connection header received
0926 16:44:24.074282 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:24.074287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:24.074371 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:24.074539 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:44:24.075214 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:24.076253 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:24.077959 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:24.078429 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:24.078577 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:24.079091 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:44:24.079228 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:44:24.079333 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:24.079489 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 16:44:24.079606 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":65}
I20260926 16:44:24.080116 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:24.072931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:34493 (local address 127.9.238.194:36643)
0926 16:44:24.073739 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:24.073761 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:24.074117 (+ 356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:24.074546 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:24.074551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:24.074562 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 16:44:24.075032 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:24.075044 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:24.076387 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:24.076395 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:24.077791 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:24.077805 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:24.078641 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:24.078649 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:24.078789 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:24.078868 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 16:44:24.079339 (+ 471us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 16:44:24.079360 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 16:44:24.079471 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 16:44:24.079669 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":5,"thread_start_us":418,"threads_started":1}
I20260926 16:44:24.093166 18236 raft_consensus.cc:1261] T 80236d3fbf4249d497b0005565d49ddf P 858926b8a54f4578b2ed598e4e94343e [term 1 FOLLOWER]: Refusing update from remote peer 53c8f79078f94da78570538801b43329: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:44:24.093119 17957 raft_consensus.cc:1261] T 80236d3fbf4249d497b0005565d49ddf P 09bb826eee674015b88848b1c2e9c54d [term 1 FOLLOWER]: Refusing update from remote peer 53c8f79078f94da78570538801b43329: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:44:24.093932 18339 consensus_queue.cc:1035] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:44:24.094537 18339 consensus_queue.cc:1035] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:44:24.177198 17793 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:44:24.177567 17793 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:44:24.181331 17793 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fcfbc557bbb54548b890bd7cc48574b4: Sending DeleteTablet for 3 replicas of tablet 80236d3fbf4249d497b0005565d49ddf
I20260926 16:44:24.182524 17932 tablet_service.cc:1514] Processing DeleteTablet for tablet 80236d3fbf4249d497b0005565d49ddf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:24 UTC) from {username='oryx', principal='oryx/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:46688
I20260926 16:44:24.182541 18071 tablet_service.cc:1514] Processing DeleteTablet for tablet 80236d3fbf4249d497b0005565d49ddf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:24 UTC) from {username='oryx', principal='oryx/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:55574
I20260926 16:44:24.183699 18212 tablet_service.cc:1514] Processing DeleteTablet for tablet 80236d3fbf4249d497b0005565d49ddf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:24 UTC) from {username='oryx', principal='oryx/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:49910
I20260926 16:44:24.194545 18375 tablet_replica.cc:330] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329: stopping tablet replica
I20260926 16:44:24.195524 18375 raft_consensus.cc:2229] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:44:24.196913 18375 raft_consensus.cc:2258] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:44:24.196982 18376 tablet_replica.cc:330] T 80236d3fbf4249d497b0005565d49ddf P 858926b8a54f4578b2ed598e4e94343e: stopping tablet replica
I20260926 16:44:24.197242 18376 raft_consensus.cc:2229] T 80236d3fbf4249d497b0005565d49ddf P 858926b8a54f4578b2ed598e4e94343e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:44:24.197510 18376 raft_consensus.cc:2258] T 80236d3fbf4249d497b0005565d49ddf P 858926b8a54f4578b2ed598e4e94343e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:44:24.199483 18375 ts_tablet_manager.cc:1905] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:44:24.206357 18376 ts_tablet_manager.cc:1905] T 80236d3fbf4249d497b0005565d49ddf P 858926b8a54f4578b2ed598e4e94343e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:44:24.207157 18375 ts_tablet_manager.cc:1918] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:44:24.207322 18375 log.cc:1196] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-1/wal/wals/80236d3fbf4249d497b0005565d49ddf
I20260926 16:44:24.207858 18375 ts_tablet_manager.cc:1939] T 80236d3fbf4249d497b0005565d49ddf P 53c8f79078f94da78570538801b43329: Deleting consensus metadata
I20260926 16:44:24.208202 18374 tablet_replica.cc:330] T 80236d3fbf4249d497b0005565d49ddf P 09bb826eee674015b88848b1c2e9c54d: stopping tablet replica
I20260926 16:44:24.208494 18374 raft_consensus.cc:2229] T 80236d3fbf4249d497b0005565d49ddf P 09bb826eee674015b88848b1c2e9c54d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:44:24.208803 18374 raft_consensus.cc:2258] T 80236d3fbf4249d497b0005565d49ddf P 09bb826eee674015b88848b1c2e9c54d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:44:24.209143 17781 catalog_manager.cc:4841] TS 53c8f79078f94da78570538801b43329 (127.9.238.194:34493): tablet 80236d3fbf4249d497b0005565d49ddf (table test-table [id=a039c22640d745d1a60eb8e2904a7b81]) successfully deleted
I20260926 16:44:24.212424 18374 ts_tablet_manager.cc:1905] T 80236d3fbf4249d497b0005565d49ddf P 09bb826eee674015b88848b1c2e9c54d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:44:24.218279 18376 ts_tablet_manager.cc:1918] T 80236d3fbf4249d497b0005565d49ddf P 858926b8a54f4578b2ed598e4e94343e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:44:24.218482 18376 log.cc:1196] T 80236d3fbf4249d497b0005565d49ddf P 858926b8a54f4578b2ed598e4e94343e: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-2/wal/wals/80236d3fbf4249d497b0005565d49ddf
I20260926 16:44:24.219033 18376 ts_tablet_manager.cc:1939] T 80236d3fbf4249d497b0005565d49ddf P 858926b8a54f4578b2ed598e4e94343e: Deleting consensus metadata
I20260926 16:44:24.220198 18374 ts_tablet_manager.cc:1918] T 80236d3fbf4249d497b0005565d49ddf P 09bb826eee674015b88848b1c2e9c54d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:44:24.220407 18374 log.cc:1196] T 80236d3fbf4249d497b0005565d49ddf P 09bb826eee674015b88848b1c2e9c54d: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-0/wal/wals/80236d3fbf4249d497b0005565d49ddf
I20260926 16:44:24.220713 17781 catalog_manager.cc:4841] TS 858926b8a54f4578b2ed598e4e94343e (127.9.238.195:34973): tablet 80236d3fbf4249d497b0005565d49ddf (table test-table [id=a039c22640d745d1a60eb8e2904a7b81]) successfully deleted
I20260926 16:44:24.220878 18374 ts_tablet_manager.cc:1939] T 80236d3fbf4249d497b0005565d49ddf P 09bb826eee674015b88848b1c2e9c54d: Deleting consensus metadata
I20260926 16:44:24.222218 17780 catalog_manager.cc:4841] TS 09bb826eee674015b88848b1c2e9c54d (127.9.238.193:41345): tablet 80236d3fbf4249d497b0005565d49ddf (table test-table [id=a039c22640d745d1a60eb8e2904a7b81]) successfully deleted
Sep 26 16:44:24 dist-test-slave-wt2j krb5kdc[17750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 16:44:24 dist-test-slave-wt2j krb5kdc[17750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:24 dist-test-slave-wt2j krb5kdc[17750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 16:44:24.274721 18384 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:24.260602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56854 (local address 127.9.238.254:41549)
0926 16:44:24.261912 (+ 1310us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:24.261918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:24.261937 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:44:24.262005 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:24.262009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:24.262145 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:24.262306 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:24.264117 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:24.265034 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:24.266575 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:24.267098 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:24.267260 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:24.274293 (+ 7033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56854: BlockingRecv error: recv got EOF from 127.0.0.1:56854 (error 108)
Metrics: {"server-negotiator.queue_time_us":1189,"thread_start_us":1032,"threads_started":1}
Sep 26 16:44:24 dist-test-slave-wt2j krb5kdc[17750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM
I20260926 16:44:24.297796 18384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:24.278875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56858 (local address 127.9.238.254:41549)
0926 16:44:24.279172 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:24.279184 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:24.279681 (+ 497us) server_negotiation.cc:407] Connection header received
0926 16:44:24.279920 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:24.279927 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:24.280022 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:24.280131 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:24.281892 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:24.282782 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:24.284736 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:24.285358 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:24.285544 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:24.291222 (+ 5678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:24.291230 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:24.291234 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:24.291318 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:24.294606 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:24.295493 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:24.295498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:24.295501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:24.295752 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:24.296177 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:24.296181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:24.296184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:24.296511 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:24.296805 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:24.297107 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 16:44:24.297368 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 16:44:24.302898 17793 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:44:24.303887 17793 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:44:24.313786 17932 tablet_service.cc:1467] Processing CreateTablet for tablet 92b3ae4f86b94ada8324a7664ae2ad7e (DEFAULT_TABLE table=test-table [id=2f5bf1ff886d41bf8916cf58513a9f45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:24.314184 18071 tablet_service.cc:1467] Processing CreateTablet for tablet 92b3ae4f86b94ada8324a7664ae2ad7e (DEFAULT_TABLE table=test-table [id=2f5bf1ff886d41bf8916cf58513a9f45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:24.314203 18212 tablet_service.cc:1467] Processing CreateTablet for tablet 92b3ae4f86b94ada8324a7664ae2ad7e (DEFAULT_TABLE table=test-table [id=2f5bf1ff886d41bf8916cf58513a9f45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:24.314453 17932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92b3ae4f86b94ada8324a7664ae2ad7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:24.314697 18212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92b3ae4f86b94ada8324a7664ae2ad7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:24.314764 18071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92b3ae4f86b94ada8324a7664ae2ad7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:24.327009 18391 tablet_bootstrap.cc:492] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d: Bootstrap starting.
I20260926 16:44:24.329779 18391 tablet_bootstrap.cc:654] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:24.330039 18390 tablet_bootstrap.cc:492] T 92b3ae4f86b94ada8324a7664ae2ad7e P 858926b8a54f4578b2ed598e4e94343e: Bootstrap starting.
I20260926 16:44:24.331987 18390 tablet_bootstrap.cc:654] T 92b3ae4f86b94ada8324a7664ae2ad7e P 858926b8a54f4578b2ed598e4e94343e: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:24.332530 18392 tablet_bootstrap.cc:492] T 92b3ae4f86b94ada8324a7664ae2ad7e P 53c8f79078f94da78570538801b43329: Bootstrap starting.
I20260926 16:44:24.335165 18392 tablet_bootstrap.cc:654] T 92b3ae4f86b94ada8324a7664ae2ad7e P 53c8f79078f94da78570538801b43329: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:24.336941 18390 tablet_bootstrap.cc:492] T 92b3ae4f86b94ada8324a7664ae2ad7e P 858926b8a54f4578b2ed598e4e94343e: No bootstrap required, opened a new log
I20260926 16:44:24.337127 18390 ts_tablet_manager.cc:1397] T 92b3ae4f86b94ada8324a7664ae2ad7e P 858926b8a54f4578b2ed598e4e94343e: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260926 16:44:24.337953 18390 raft_consensus.cc:348] T 92b3ae4f86b94ada8324a7664ae2ad7e P 858926b8a54f4578b2ed598e4e94343e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:24.338140 18392 tablet_bootstrap.cc:492] T 92b3ae4f86b94ada8324a7664ae2ad7e P 53c8f79078f94da78570538801b43329: No bootstrap required, opened a new log
I20260926 16:44:24.338238 18390 raft_consensus.cc:374] T 92b3ae4f86b94ada8324a7664ae2ad7e P 858926b8a54f4578b2ed598e4e94343e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:24.338340 18390 raft_consensus.cc:729] T 92b3ae4f86b94ada8324a7664ae2ad7e P 858926b8a54f4578b2ed598e4e94343e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 858926b8a54f4578b2ed598e4e94343e, State: Initialized, Role: FOLLOWER
I20260926 16:44:24.338376 18392 ts_tablet_manager.cc:1397] T 92b3ae4f86b94ada8324a7664ae2ad7e P 53c8f79078f94da78570538801b43329: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 16:44:24.339308 18391 tablet_bootstrap.cc:492] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d: No bootstrap required, opened a new log
I20260926 16:44:24.339473 18391 ts_tablet_manager.cc:1397] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20260926 16:44:24.339591 18392 raft_consensus.cc:348] T 92b3ae4f86b94ada8324a7664ae2ad7e P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:24.339900 18392 raft_consensus.cc:374] T 92b3ae4f86b94ada8324a7664ae2ad7e P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:24.340022 18392 raft_consensus.cc:729] T 92b3ae4f86b94ada8324a7664ae2ad7e P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c8f79078f94da78570538801b43329, State: Initialized, Role: FOLLOWER
I20260926 16:44:24.340356 18391 raft_consensus.cc:348] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:24.340387 18392 consensus_queue.cc:260] T 92b3ae4f86b94ada8324a7664ae2ad7e P 53c8f79078f94da78570538801b43329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:24.340632 18391 raft_consensus.cc:374] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:24.340729 18391 raft_consensus.cc:729] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09bb826eee674015b88848b1c2e9c54d, State: Initialized, Role: FOLLOWER
I20260926 16:44:24.341074 18390 consensus_queue.cc:260] T 92b3ae4f86b94ada8324a7664ae2ad7e P 858926b8a54f4578b2ed598e4e94343e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:24.341264 18392 ts_tablet_manager.cc:1428] T 92b3ae4f86b94ada8324a7664ae2ad7e P 53c8f79078f94da78570538801b43329: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 16:44:24.341666 18390 ts_tablet_manager.cc:1428] T 92b3ae4f86b94ada8324a7664ae2ad7e P 858926b8a54f4578b2ed598e4e94343e: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260926 16:44:24.341950 18391 consensus_queue.cc:260] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:24.342680 18391 ts_tablet_manager.cc:1428] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 16:44:24.377612 18314 raft_consensus.cc:482] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:44:24.378037 18314 raft_consensus.cc:504] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:24.380592 18314 leader_election.cc:288] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53c8f79078f94da78570538801b43329 (127.9.238.194:34493), 858926b8a54f4578b2ed598e4e94343e (127.9.238.195:34973)
I20260926 16:44:24.395124 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:24.381225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:41833 (local address 127.9.238.194:34493)
0926 16:44:24.381532 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:24.381543 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:24.382097 (+ 554us) server_negotiation.cc:407] Connection header received
0926 16:44:24.382551 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:24.382564 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:24.382712 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:24.382823 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:24.384254 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:24.386138 (+ 1884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:24.390814 (+ 4676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:24.393366 (+ 2552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:24.393643 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:24.393750 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:24.394514 (+ 764us) server_negotiation.cc:299] Negotiation successful
0926 16:44:24.394717 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 16:44:24.396284 18396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:24.380885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:34493 (local address 127.9.238.193:41833)
0926 16:44:24.381939 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:24.381967 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:24.382281 (+ 314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:24.383177 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:24.383202 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:24.383248 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:24.383924 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:24.383955 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:24.386489 (+ 2534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:24.386500 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:24.390229 (+ 3729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:24.390471 (+ 242us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:24.393979 (+ 3508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:24.393988 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:24.394249 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:24.394302 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 16:44:24.395520 (+ 1218us) client_negotiation.cc:240] Negotiation successful
0926 16:44:24.395763 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"thread_start_us":668,"threads_started":1}
I20260926 16:44:24.398415 18092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b3ae4f86b94ada8324a7664ae2ad7e" candidate_uuid: "09bb826eee674015b88848b1c2e9c54d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c8f79078f94da78570538801b43329" is_pre_election: true
I20260926 16:44:24.398919 18092 raft_consensus.cc:2454] T 92b3ae4f86b94ada8324a7664ae2ad7e P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09bb826eee674015b88848b1c2e9c54d in term 0.
I20260926 16:44:24.399829 17884 leader_election.cc:302] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09bb826eee674015b88848b1c2e9c54d, 53c8f79078f94da78570538801b43329; no voters:
I20260926 16:44:24.400424 18314 raft_consensus.cc:2781] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:44:24.400612 18314 raft_consensus.cc:482] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:44:24.400759 18314 raft_consensus.cc:3037] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:24.404512 18314 raft_consensus.cc:504] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:24.405575 18314 leader_election.cc:288] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d [CANDIDATE]: Term 1 election: Requested vote from peers 53c8f79078f94da78570538801b43329 (127.9.238.194:34493), 858926b8a54f4578b2ed598e4e94343e (127.9.238.195:34973)
I20260926 16:44:24.406172 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:24.382245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:40613 (local address 127.9.238.195:34973)
0926 16:44:24.397624 (+ 15379us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:24.397632 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:24.397663 (+ 31us) server_negotiation.cc:407] Connection header received
0926 16:44:24.397787 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:24.397794 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:24.397899 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:24.398120 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:24.398735 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:24.400144 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:24.402888 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:24.404553 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:24.404997 (+ 444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:24.405110 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:24.405367 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 16:44:24.405568 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15033,"thread_start_us":15132,"threads_started":1}
I20260926 16:44:24.406267 18396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:24.382055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:34973 (local address 127.9.238.193:40613)
0926 16:44:24.396644 (+ 14589us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:24.396690 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:24.396867 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:24.398202 (+ 1335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:24.398209 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:24.398256 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:24.398499 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:24.398518 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:24.400343 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:24.400350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:24.402597 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:24.402643 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:24.404846 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:24.404855 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:24.405081 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:24.405125 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 16:44:24.405249 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 16:44:24.405410 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14445,"thread_start_us":15188,"threads_started":1}
I20260926 16:44:24.406504 18092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b3ae4f86b94ada8324a7664ae2ad7e" candidate_uuid: "09bb826eee674015b88848b1c2e9c54d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c8f79078f94da78570538801b43329"
I20260926 16:44:24.406836 18092 raft_consensus.cc:3037] T 92b3ae4f86b94ada8324a7664ae2ad7e P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:24.407343 18235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b3ae4f86b94ada8324a7664ae2ad7e" candidate_uuid: "09bb826eee674015b88848b1c2e9c54d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "858926b8a54f4578b2ed598e4e94343e" is_pre_election: true
I20260926 16:44:24.407761 18235 raft_consensus.cc:2454] T 92b3ae4f86b94ada8324a7664ae2ad7e P 858926b8a54f4578b2ed598e4e94343e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09bb826eee674015b88848b1c2e9c54d in term 0.
I20260926 16:44:24.410357 18092 raft_consensus.cc:2454] T 92b3ae4f86b94ada8324a7664ae2ad7e P 53c8f79078f94da78570538801b43329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09bb826eee674015b88848b1c2e9c54d in term 1.
I20260926 16:44:24.410815 18236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b3ae4f86b94ada8324a7664ae2ad7e" candidate_uuid: "09bb826eee674015b88848b1c2e9c54d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "858926b8a54f4578b2ed598e4e94343e"
I20260926 16:44:24.411120 18236 raft_consensus.cc:3037] T 92b3ae4f86b94ada8324a7664ae2ad7e P 858926b8a54f4578b2ed598e4e94343e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:24.411118 17884 leader_election.cc:302] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09bb826eee674015b88848b1c2e9c54d, 53c8f79078f94da78570538801b43329; no voters:
I20260926 16:44:24.411540 18314 raft_consensus.cc:2781] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:44:24.413300 18314 raft_consensus.cc:686] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d [term 1 LEADER]: Becoming Leader. State: Replica: 09bb826eee674015b88848b1c2e9c54d, State: Running, Role: LEADER
I20260926 16:44:24.413861 18314 consensus_queue.cc:237] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:24.414397 18236 raft_consensus.cc:2454] T 92b3ae4f86b94ada8324a7664ae2ad7e P 858926b8a54f4578b2ed598e4e94343e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09bb826eee674015b88848b1c2e9c54d in term 1.
I20260926 16:44:24.418968 17794 catalog_manager.cc:5495] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d reported cstate change: term changed from 0 to 1, leader changed from <none> to 09bb826eee674015b88848b1c2e9c54d (127.9.238.193). New cstate: current_term: 1 leader_uuid: "09bb826eee674015b88848b1c2e9c54d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } health_report { overall_health: UNKNOWN } } }
I20260926 16:44:24.450776 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:24.440771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46700 (local address 127.9.238.193:41345)
0926 16:44:24.441968 (+ 1197us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:24.441975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:24.442012 (+ 37us) server_negotiation.cc:407] Connection header received
0926 16:44:24.442140 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:24.442146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:24.442265 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:24.442474 (+ 209us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:44:24.443106 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:24.444479 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:24.446861 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:24.447711 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:24.447858 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:24.448638 (+ 780us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:44:24.449754 (+ 1116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:44:24.449935 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:24.450161 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 16:44:24.450286 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":985,"thread_start_us":766,"threads_started":1}
I20260926 16:44:24.456064 18236 raft_consensus.cc:1261] T 92b3ae4f86b94ada8324a7664ae2ad7e P 858926b8a54f4578b2ed598e4e94343e [term 1 FOLLOWER]: Refusing update from remote peer 09bb826eee674015b88848b1c2e9c54d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:44:24.456157 18092 raft_consensus.cc:1261] T 92b3ae4f86b94ada8324a7664ae2ad7e P 53c8f79078f94da78570538801b43329 [term 1 FOLLOWER]: Refusing update from remote peer 09bb826eee674015b88848b1c2e9c54d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:44:24.457063 18399 consensus_queue.cc:1035] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d [LEADER]: Connected to new peer: Peer: permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:44:24.457528 18314 consensus_queue.cc:1035] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d [LEADER]: Connected to new peer: Peer: permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:44:24.470371 18357 mvcc.cc:205] Tried to move back new op lower bound from 7333646600002330624 to 7333646599845298176. Current Snapshot: MvccSnapshot[applied={T|T < 7333646600002330624 or (T in {7333646600002330624})}]
I20260926 16:44:24.481549 17793 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:44:24.481832 17793 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:44:24.487304 17793 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fcfbc557bbb54548b890bd7cc48574b4: Sending DeleteTablet for 3 replicas of tablet 92b3ae4f86b94ada8324a7664ae2ad7e
I20260926 16:44:24.488430 17932 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b3ae4f86b94ada8324a7664ae2ad7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:24 UTC) from {username='oryx', principal='oryx/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:46688
I20260926 16:44:24.488845 18374 tablet_replica.cc:330] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d: stopping tablet replica
I20260926 16:44:24.489108 18374 raft_consensus.cc:2229] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:44:24.489662 18374 raft_consensus.cc:2258] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:44:24.489985 18212 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b3ae4f86b94ada8324a7664ae2ad7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:24 UTC) from {username='oryx', principal='oryx/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:49910
I20260926 16:44:24.490005 18071 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b3ae4f86b94ada8324a7664ae2ad7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:24 UTC) from {username='oryx', principal='oryx/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:55574
I20260926 16:44:24.490365 18376 tablet_replica.cc:330] T 92b3ae4f86b94ada8324a7664ae2ad7e P 858926b8a54f4578b2ed598e4e94343e: stopping tablet replica
I20260926 16:44:24.490406 18375 tablet_replica.cc:330] T 92b3ae4f86b94ada8324a7664ae2ad7e P 53c8f79078f94da78570538801b43329: stopping tablet replica
I20260926 16:44:24.490598 18376 raft_consensus.cc:2229] T 92b3ae4f86b94ada8324a7664ae2ad7e P 858926b8a54f4578b2ed598e4e94343e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:44:24.490626 18375 raft_consensus.cc:2229] T 92b3ae4f86b94ada8324a7664ae2ad7e P 53c8f79078f94da78570538801b43329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:44:24.490830 18376 raft_consensus.cc:2258] T 92b3ae4f86b94ada8324a7664ae2ad7e P 858926b8a54f4578b2ed598e4e94343e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:44:24.490856 18375 raft_consensus.cc:2258] T 92b3ae4f86b94ada8324a7664ae2ad7e P 53c8f79078f94da78570538801b43329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:44:24.492398 17793 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 16:44:24.493046 18376 ts_tablet_manager.cc:1905] T 92b3ae4f86b94ada8324a7664ae2ad7e P 858926b8a54f4578b2ed598e4e94343e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:44:24.493146 18374 ts_tablet_manager.cc:1905] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 16:44:24.493824 17793 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:44:24.498459 18375 ts_tablet_manager.cc:1905] T 92b3ae4f86b94ada8324a7664ae2ad7e P 53c8f79078f94da78570538801b43329: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:44:24.502470 18374 ts_tablet_manager.cc:1918] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:44:24.502687 18374 log.cc:1196] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-0/wal/wals/92b3ae4f86b94ada8324a7664ae2ad7e
I20260926 16:44:24.503149 18374 ts_tablet_manager.cc:1939] T 92b3ae4f86b94ada8324a7664ae2ad7e P 09bb826eee674015b88848b1c2e9c54d: Deleting consensus metadata
I20260926 16:44:24.505357 17932 tablet_service.cc:1467] Processing CreateTablet for tablet 6ede01dfd47941df9661aa29e29aef1e (DEFAULT_TABLE table=test-table [id=5aa5d6c281194f27b631f1a530198d8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:24.505930 17932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ede01dfd47941df9661aa29e29aef1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:24.505820 18212 tablet_service.cc:1467] Processing CreateTablet for tablet 6ede01dfd47941df9661aa29e29aef1e (DEFAULT_TABLE table=test-table [id=5aa5d6c281194f27b631f1a530198d8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:24.506426 18212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ede01dfd47941df9661aa29e29aef1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:24.512794 18071 tablet_service.cc:1467] Processing CreateTablet for tablet 6ede01dfd47941df9661aa29e29aef1e (DEFAULT_TABLE table=test-table [id=5aa5d6c281194f27b631f1a530198d8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:24.513533 18071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ede01dfd47941df9661aa29e29aef1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:24.517551 18376 ts_tablet_manager.cc:1918] T 92b3ae4f86b94ada8324a7664ae2ad7e P 858926b8a54f4578b2ed598e4e94343e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:44:24.517755 18376 log.cc:1196] T 92b3ae4f86b94ada8324a7664ae2ad7e P 858926b8a54f4578b2ed598e4e94343e: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-2/wal/wals/92b3ae4f86b94ada8324a7664ae2ad7e
I20260926 16:44:24.518345 18376 ts_tablet_manager.cc:1939] T 92b3ae4f86b94ada8324a7664ae2ad7e P 858926b8a54f4578b2ed598e4e94343e: Deleting consensus metadata
I20260926 16:44:24.519935 17781 catalog_manager.cc:4841] TS 858926b8a54f4578b2ed598e4e94343e (127.9.238.195:34973): tablet 92b3ae4f86b94ada8324a7664ae2ad7e (table test-table [id=2f5bf1ff886d41bf8916cf58513a9f45]) successfully deleted
I20260926 16:44:24.522857 18390 tablet_bootstrap.cc:492] T 6ede01dfd47941df9661aa29e29aef1e P 858926b8a54f4578b2ed598e4e94343e: Bootstrap starting.
I20260926 16:44:24.523540 17780 catalog_manager.cc:4841] TS 09bb826eee674015b88848b1c2e9c54d (127.9.238.193:41345): tablet 92b3ae4f86b94ada8324a7664ae2ad7e (table test-table [id=2f5bf1ff886d41bf8916cf58513a9f45]) successfully deleted
I20260926 16:44:24.524372 18392 tablet_bootstrap.cc:492] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329: Bootstrap starting.
I20260926 16:44:24.525099 18390 tablet_bootstrap.cc:654] T 6ede01dfd47941df9661aa29e29aef1e P 858926b8a54f4578b2ed598e4e94343e: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:24.526750 18392 tablet_bootstrap.cc:654] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:24.527139 18391 tablet_bootstrap.cc:492] T 6ede01dfd47941df9661aa29e29aef1e P 09bb826eee674015b88848b1c2e9c54d: Bootstrap starting.
I20260926 16:44:24.529078 18391 tablet_bootstrap.cc:654] T 6ede01dfd47941df9661aa29e29aef1e P 09bb826eee674015b88848b1c2e9c54d: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:24.537742 18375 ts_tablet_manager.cc:1918] T 92b3ae4f86b94ada8324a7664ae2ad7e P 53c8f79078f94da78570538801b43329: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:44:24.538069 18391 tablet_bootstrap.cc:492] T 6ede01dfd47941df9661aa29e29aef1e P 09bb826eee674015b88848b1c2e9c54d: No bootstrap required, opened a new log
I20260926 16:44:24.538288 18391 ts_tablet_manager.cc:1397] T 6ede01dfd47941df9661aa29e29aef1e P 09bb826eee674015b88848b1c2e9c54d: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260926 16:44:24.538497 18375 log.cc:1196] T 92b3ae4f86b94ada8324a7664ae2ad7e P 53c8f79078f94da78570538801b43329: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-1/wal/wals/92b3ae4f86b94ada8324a7664ae2ad7e
I20260926 16:44:24.539121 18375 ts_tablet_manager.cc:1939] T 92b3ae4f86b94ada8324a7664ae2ad7e P 53c8f79078f94da78570538801b43329: Deleting consensus metadata
I20260926 16:44:24.539242 18391 raft_consensus.cc:348] T 6ede01dfd47941df9661aa29e29aef1e P 09bb826eee674015b88848b1c2e9c54d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:24.539364 18390 tablet_bootstrap.cc:492] T 6ede01dfd47941df9661aa29e29aef1e P 858926b8a54f4578b2ed598e4e94343e: No bootstrap required, opened a new log
I20260926 16:44:24.539536 18391 raft_consensus.cc:374] T 6ede01dfd47941df9661aa29e29aef1e P 09bb826eee674015b88848b1c2e9c54d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:24.539651 18391 raft_consensus.cc:729] T 6ede01dfd47941df9661aa29e29aef1e P 09bb826eee674015b88848b1c2e9c54d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09bb826eee674015b88848b1c2e9c54d, State: Initialized, Role: FOLLOWER
I20260926 16:44:24.539734 18390 ts_tablet_manager.cc:1397] T 6ede01dfd47941df9661aa29e29aef1e P 858926b8a54f4578b2ed598e4e94343e: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.012s
I20260926 16:44:24.539954 18391 consensus_queue.cc:260] T 6ede01dfd47941df9661aa29e29aef1e P 09bb826eee674015b88848b1c2e9c54d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:24.540572 18391 ts_tablet_manager.cc:1428] T 6ede01dfd47941df9661aa29e29aef1e P 09bb826eee674015b88848b1c2e9c54d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 16:44:24.540634 17781 catalog_manager.cc:4841] TS 53c8f79078f94da78570538801b43329 (127.9.238.194:34493): tablet 92b3ae4f86b94ada8324a7664ae2ad7e (table test-table [id=2f5bf1ff886d41bf8916cf58513a9f45]) successfully deleted
I20260926 16:44:24.541680 18392 tablet_bootstrap.cc:492] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329: No bootstrap required, opened a new log
I20260926 16:44:24.541893 18392 ts_tablet_manager.cc:1397] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.006s
I20260926 16:44:24.542946 18392 raft_consensus.cc:348] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:24.543298 18392 raft_consensus.cc:374] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:24.543406 18392 raft_consensus.cc:729] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c8f79078f94da78570538801b43329, State: Initialized, Role: FOLLOWER
I20260926 16:44:24.544305 18392 consensus_queue.cc:260] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:24.544997 18392 ts_tablet_manager.cc:1428] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260926 16:44:24.546182 18390 raft_consensus.cc:348] T 6ede01dfd47941df9661aa29e29aef1e P 858926b8a54f4578b2ed598e4e94343e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:24.546504 18390 raft_consensus.cc:374] T 6ede01dfd47941df9661aa29e29aef1e P 858926b8a54f4578b2ed598e4e94343e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:24.546612 18390 raft_consensus.cc:729] T 6ede01dfd47941df9661aa29e29aef1e P 858926b8a54f4578b2ed598e4e94343e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 858926b8a54f4578b2ed598e4e94343e, State: Initialized, Role: FOLLOWER
I20260926 16:44:24.547001 18390 consensus_queue.cc:260] T 6ede01dfd47941df9661aa29e29aef1e P 858926b8a54f4578b2ed598e4e94343e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:24.547719 18390 ts_tablet_manager.cc:1428] T 6ede01dfd47941df9661aa29e29aef1e P 858926b8a54f4578b2ed598e4e94343e: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20260926 16:44:24.662124 18372 raft_consensus.cc:482] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:44:24.662456 18372 raft_consensus.cc:504] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:24.663470 18372 leader_election.cc:288] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09bb826eee674015b88848b1c2e9c54d (127.9.238.193:41345), 858926b8a54f4578b2ed598e4e94343e (127.9.238.195:34973)
I20260926 16:44:24.664511 18236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ede01dfd47941df9661aa29e29aef1e" candidate_uuid: "53c8f79078f94da78570538801b43329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "858926b8a54f4578b2ed598e4e94343e" is_pre_election: true
I20260926 16:44:24.664467 17956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ede01dfd47941df9661aa29e29aef1e" candidate_uuid: "53c8f79078f94da78570538801b43329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09bb826eee674015b88848b1c2e9c54d" is_pre_election: true
I20260926 16:44:24.664867 18236 raft_consensus.cc:2454] T 6ede01dfd47941df9661aa29e29aef1e P 858926b8a54f4578b2ed598e4e94343e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53c8f79078f94da78570538801b43329 in term 0.
I20260926 16:44:24.664865 17956 raft_consensus.cc:2454] T 6ede01dfd47941df9661aa29e29aef1e P 09bb826eee674015b88848b1c2e9c54d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53c8f79078f94da78570538801b43329 in term 0.
I20260926 16:44:24.665504 18024 leader_election.cc:302] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53c8f79078f94da78570538801b43329, 858926b8a54f4578b2ed598e4e94343e; no voters:
I20260926 16:44:24.666203 18372 raft_consensus.cc:2781] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:44:24.666347 18372 raft_consensus.cc:482] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:44:24.666463 18372 raft_consensus.cc:3037] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:24.670202 18372 raft_consensus.cc:504] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:24.671902 17956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ede01dfd47941df9661aa29e29aef1e" candidate_uuid: "53c8f79078f94da78570538801b43329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09bb826eee674015b88848b1c2e9c54d"
I20260926 16:44:24.671931 18236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ede01dfd47941df9661aa29e29aef1e" candidate_uuid: "53c8f79078f94da78570538801b43329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "858926b8a54f4578b2ed598e4e94343e"
I20260926 16:44:24.672197 17956 raft_consensus.cc:3037] T 6ede01dfd47941df9661aa29e29aef1e P 09bb826eee674015b88848b1c2e9c54d [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:24.672230 18236 raft_consensus.cc:3037] T 6ede01dfd47941df9661aa29e29aef1e P 858926b8a54f4578b2ed598e4e94343e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:24.675932 18236 raft_consensus.cc:2454] T 6ede01dfd47941df9661aa29e29aef1e P 858926b8a54f4578b2ed598e4e94343e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53c8f79078f94da78570538801b43329 in term 1.
I20260926 16:44:24.676630 18024 leader_election.cc:302] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53c8f79078f94da78570538801b43329, 858926b8a54f4578b2ed598e4e94343e; no voters:
I20260926 16:44:24.676621 17956 raft_consensus.cc:2454] T 6ede01dfd47941df9661aa29e29aef1e P 09bb826eee674015b88848b1c2e9c54d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53c8f79078f94da78570538801b43329 in term 1.
I20260926 16:44:24.677599 18372 leader_election.cc:288] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329 [CANDIDATE]: Term 1 election: Requested vote from peers 09bb826eee674015b88848b1c2e9c54d (127.9.238.193:41345), 858926b8a54f4578b2ed598e4e94343e (127.9.238.195:34973)
I20260926 16:44:24.677949 18372 raft_consensus.cc:2781] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:44:24.678174 18372 raft_consensus.cc:686] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329 [term 1 LEADER]: Becoming Leader. State: Replica: 53c8f79078f94da78570538801b43329, State: Running, Role: LEADER
I20260926 16:44:24.678560 18372 consensus_queue.cc:237] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } }
I20260926 16:44:24.691154 17794 catalog_manager.cc:5495] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53c8f79078f94da78570538801b43329 (127.9.238.194). New cstate: current_term: 1 leader_uuid: "53c8f79078f94da78570538801b43329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53c8f79078f94da78570538801b43329" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 34493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 } health_report { overall_health: UNKNOWN } } }
I20260926 16:44:24.746503 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:24.738723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55618 (local address 127.9.238.194:34493)
0926 16:44:24.738953 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:24.738959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:24.738979 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:44:24.739071 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:24.739077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:24.739171 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:24.739273 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:44:24.739947 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:24.741156 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:24.743336 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:24.744217 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:24.744354 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:24.745034 (+ 680us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:44:24.745261 (+ 227us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:44:24.745359 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:24.746002 (+ 643us) server_negotiation.cc:299] Negotiation successful
0926 16:44:24.746108 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 16:44:24.772183 18235 raft_consensus.cc:1261] T 6ede01dfd47941df9661aa29e29aef1e P 858926b8a54f4578b2ed598e4e94343e [term 1 FOLLOWER]: Refusing update from remote peer 53c8f79078f94da78570538801b43329: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:44:24.773191 18409 consensus_queue.cc:1035] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "858926b8a54f4578b2ed598e4e94343e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 34973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:44:24.776775 17956 raft_consensus.cc:1261] T 6ede01dfd47941df9661aa29e29aef1e P 09bb826eee674015b88848b1c2e9c54d [term 1 FOLLOWER]: Refusing update from remote peer 53c8f79078f94da78570538801b43329: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:44:24.777704 18411 consensus_queue.cc:1035] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09bb826eee674015b88848b1c2e9c54d" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:44:24.791622 18358 mvcc.cc:205] Tried to move back new op lower bound from 7333646601300344832 to 7333646600927076352. Current Snapshot: MvccSnapshot[applied={T|T < 7333646601300344832}]
I20260926 16:44:24.889068 17794 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:44:24.889406 17794 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:44:24.894105 17794 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fcfbc557bbb54548b890bd7cc48574b4: Sending DeleteTablet for 3 replicas of tablet 6ede01dfd47941df9661aa29e29aef1e
I20260926 16:44:24.895335 17932 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ede01dfd47941df9661aa29e29aef1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:24 UTC) from {username='oryx', principal='oryx/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:46688
I20260926 16:44:24.895794 18374 tablet_replica.cc:330] T 6ede01dfd47941df9661aa29e29aef1e P 09bb826eee674015b88848b1c2e9c54d: stopping tablet replica
I20260926 16:44:24.896005 18374 raft_consensus.cc:2229] T 6ede01dfd47941df9661aa29e29aef1e P 09bb826eee674015b88848b1c2e9c54d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:44:24.896265 18374 raft_consensus.cc:2258] T 6ede01dfd47941df9661aa29e29aef1e P 09bb826eee674015b88848b1c2e9c54d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:44:24.897043 18212 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ede01dfd47941df9661aa29e29aef1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:24 UTC) from {username='oryx', principal='oryx/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:49910
I20260926 16:44:24.897361 18376 tablet_replica.cc:330] T 6ede01dfd47941df9661aa29e29aef1e P 858926b8a54f4578b2ed598e4e94343e: stopping tablet replica
I20260926 16:44:24.897557 18376 raft_consensus.cc:2229] T 6ede01dfd47941df9661aa29e29aef1e P 858926b8a54f4578b2ed598e4e94343e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:44:24.897835 18376 raft_consensus.cc:2258] T 6ede01dfd47941df9661aa29e29aef1e P 858926b8a54f4578b2ed598e4e94343e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:44:24.899610 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 17866
I20260926 16:44:24.901058 18374 ts_tablet_manager.cc:1905] T 6ede01dfd47941df9661aa29e29aef1e P 09bb826eee674015b88848b1c2e9c54d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:44:24.903949 18376 ts_tablet_manager.cc:1905] T 6ede01dfd47941df9661aa29e29aef1e P 858926b8a54f4578b2ed598e4e94343e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:44:24.913331 18374 ts_tablet_manager.cc:1918] T 6ede01dfd47941df9661aa29e29aef1e P 09bb826eee674015b88848b1c2e9c54d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:44:24.913520 18374 log.cc:1196] T 6ede01dfd47941df9661aa29e29aef1e P 09bb826eee674015b88848b1c2e9c54d: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-0/wal/wals/6ede01dfd47941df9661aa29e29aef1e
I20260926 16:44:24.914059 18374 ts_tablet_manager.cc:1939] T 6ede01dfd47941df9661aa29e29aef1e P 09bb826eee674015b88848b1c2e9c54d: Deleting consensus metadata
I20260926 16:44:24.915374 17780 catalog_manager.cc:4841] TS 09bb826eee674015b88848b1c2e9c54d (127.9.238.193:41345): tablet 6ede01dfd47941df9661aa29e29aef1e (table test-table [id=5aa5d6c281194f27b631f1a530198d8d]) successfully deleted
I20260926 16:44:24.916507 18071 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ede01dfd47941df9661aa29e29aef1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:24 UTC) from {username='oryx', principal='oryx/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:55574
I20260926 16:44:24.917037 18375 tablet_replica.cc:330] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329: stopping tablet replica
I20260926 16:44:24.917351 18375 raft_consensus.cc:2229] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:44:24.917977 18375 raft_consensus.cc:2258] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:44:24.920004 18375 ts_tablet_manager.cc:1905] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:44:24.925024 18376 ts_tablet_manager.cc:1918] T 6ede01dfd47941df9661aa29e29aef1e P 858926b8a54f4578b2ed598e4e94343e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:44:24.925225 18376 log.cc:1196] T 6ede01dfd47941df9661aa29e29aef1e P 858926b8a54f4578b2ed598e4e94343e: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-2/wal/wals/6ede01dfd47941df9661aa29e29aef1e
I20260926 16:44:24.925653 18376 ts_tablet_manager.cc:1939] T 6ede01dfd47941df9661aa29e29aef1e P 858926b8a54f4578b2ed598e4e94343e: Deleting consensus metadata
I20260926 16:44:24.926941 17781 catalog_manager.cc:4841] TS 858926b8a54f4578b2ed598e4e94343e (127.9.238.195:34973): tablet 6ede01dfd47941df9661aa29e29aef1e (table test-table [id=5aa5d6c281194f27b631f1a530198d8d]) successfully deleted
I20260926 16:44:24.930929 18375 ts_tablet_manager.cc:1918] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 16:44:24.931099 18375 log.cc:1196] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790440988577176-10171-0/minicluster-data/ts-1/wal/wals/6ede01dfd47941df9661aa29e29aef1e
I20260926 16:44:24.931639 18375 ts_tablet_manager.cc:1939] T 6ede01dfd47941df9661aa29e29aef1e P 53c8f79078f94da78570538801b43329: Deleting consensus metadata
I20260926 16:44:24.933122 17781 catalog_manager.cc:4841] TS 53c8f79078f94da78570538801b43329 (127.9.238.194:34493): tablet 6ede01dfd47941df9661aa29e29aef1e (table test-table [id=5aa5d6c281194f27b631f1a530198d8d]) successfully deleted
Sep 26 16:44:24 dist-test-slave-wt2j krb5kdc[17750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.238.193@KRBTEST.COM
I20260926 16:44:24.958981 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:24.901625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46706 (local address 127.9.238.193:41345)
0926 16:44:24.901850 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:24.901857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:24.928039 (+ 26182us) server_negotiation.cc:407] Connection header received
0926 16:44:24.928284 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:24.928292 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:24.928430 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:24.928615 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:24.930954 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:24.932349 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:24.935530 (+ 3181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:24.936419 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:24.936624 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:24.949158 (+ 12534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:24.949189 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:24.949217 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:24.949283 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:24.954416 (+ 5133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:24.955714 (+ 1298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:24.955727 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:24.955731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:24.955956 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:24.956519 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:24.956527 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:24.956531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:24.957165 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:24.957691 (+ 526us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:24.958089 (+ 398us) server_negotiation.cc:299] Negotiation successful
0926 16:44:24.958441 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 16:44:24.960402 17997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:25.126938 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 17866
I20260926 16:44:25.150470 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 18006
Sep 26 16:44:25 dist-test-slave-wt2j krb5kdc[17750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.238.194@KRBTEST.COM
I20260926 16:44:25.170888 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:25.151776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55632 (local address 127.9.238.194:34493)
0926 16:44:25.152026 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:25.152036 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:25.152059 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:44:25.152329 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:25.152335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:25.152433 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:25.152601 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:25.153984 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:25.154997 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:25.157068 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:25.157658 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:25.157894 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:25.163195 (+ 5301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:25.163219 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:25.163242 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:25.163309 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:25.167079 (+ 3770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:25.168168 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:25.168178 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:25.168181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:25.168456 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:25.169082 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:25.169088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:25.169091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:25.169641 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:25.169932 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:25.170264 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 16:44:25.170562 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260926 16:44:25.171993 18137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:25.328341 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 18006
W20260926 16:44:25.361891 18165 connection.cc:535] server connection from 127.9.238.194:54259 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 16:44:25.362452 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 18146
Sep 26 16:44:25 dist-test-slave-wt2j krb5kdc[17750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.9.238.195@KRBTEST.COM
I20260926 16:44:25.382118 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:25.363286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49924 (local address 127.9.238.195:34973)
0926 16:44:25.363817 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:25.363821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:25.363841 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:44:25.363899 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:25.363903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:25.363966 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:25.364072 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:25.365949 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:25.366847 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:25.368738 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:25.369173 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:25.369292 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:25.375272 (+ 5980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:25.375289 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:25.375307 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:25.375354 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:25.378612 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:25.379802 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:25.379810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:25.379812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:25.379964 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:25.380513 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:25.380517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:25.380520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:25.380906 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:25.381166 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:25.381584 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 16:44:25.381768 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":312,"threads_started":1}
I20260926 16:44:25.382789 18277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:25.528877 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 18146
I20260926 16:44:25.554255 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 17763
I20260926 16:44:25.555971 17835 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:25.676546 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 17763
2026-09-26T16:44:25Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7958 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:44:25 dist-test-slave-wt2j krb5kdc[18427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:44:25 dist-test-slave-wt2j krb5kdc[18427](info): set up 2 sockets
Sep 26 16:44:25 dist-test-slave-wt2j krb5kdc[18427](info): commencing operation
krb5kdc: starting...
W20260926 16:44:26.817833 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:44:26 dist-test-slave-wt2j krb5kdc[18427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:44:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:44:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:44:27.092350 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:43569
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:40783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:43569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:27.506769 18440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:27.507203 18440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:27.507381 18440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:27.515889 18440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:44:27.516050 18440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:27.516142 18440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:27.516258 18440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:44:27.516367 18440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:44:27.528082 18440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:40783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:43569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:43569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:27.529946 18440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:27.531657 18440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:27.547034 18445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:27.547056 18448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:27.546965 18446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:27.547621 18440 server_base.cc:1034] running on GCE node
I20260926 16:44:27.548682 18440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:27.550545 18440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:27.551807 18440 hybrid_clock.cc:648] HybridClock initialized: now 1790441067551758 us; error 38 us; skew 500 ppm
I20260926 16:44:27.552327 18440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T16:44:27Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1942 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:44:27 dist-test-slave-wt2j krb5kdc[18454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:44:27 dist-test-slave-wt2j krb5kdc[18454](info): set up 2 sockets
Sep 26 16:44:27 dist-test-slave-wt2j krb5kdc[18454](info): commencing operation
krb5kdc: starting...
W20260926 16:44:28.766110 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:44:28 dist-test-slave-wt2j krb5kdc[18454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:44:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:44:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:44:29.040486 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:35741
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:37971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:35741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:29.398576 18467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:29.398900 18467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:29.399071 18467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:29.406558 18467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:44:29.406733 18467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:29.406843 18467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:29.406955 18467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:44:29.407045 18467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:44:29.417327 18467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:37971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:35741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:35741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:29.418839 18467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:29.420264 18467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:29.433405 18472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:29.433696 18473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:29.435982 18467 server_base.cc:1034] running on GCE node
W20260926 16:44:29.435976 18475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:29.437426 18467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:29.439632 18467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:29.440953 18467 hybrid_clock.cc:648] HybridClock initialized: now 1790441069440904 us; error 72 us; skew 500 ppm
I20260926 16:44:29.441452 18467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:29 dist-test-slave-wt2j krb5kdc[18454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441069, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:29.445890 18467 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:44:29.448297 18467 webserver.cc:460] Webserver started at http://127.9.238.254:35863/ using document root <none> and password file <none>
I20260926 16:44:29.449331 18467 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:29.449508 18467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:29.449911 18467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:29.454068 18467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "ac634b3bed1740af968cd1ca61c511fd"
format_stamp: "Formatted at 2026-09-26 16:44:29 on dist-test-slave-wt2j"
server_key: "1605550bb4d267bd2d27e46d558c5cae"
server_key_iv: "23e20522db46fe437e5088428c0ca594"
server_key_version: "encryptionkey@0"
I20260926 16:44:29.455070 18467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "ac634b3bed1740af968cd1ca61c511fd"
format_stamp: "Formatted at 2026-09-26 16:44:29 on dist-test-slave-wt2j"
server_key: "1605550bb4d267bd2d27e46d558c5cae"
server_key_iv: "23e20522db46fe437e5088428c0ca594"
server_key_version: "encryptionkey@0"
I20260926 16:44:29.462349 18467 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260926 16:44:29.466809 18482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:29.468796 18467 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:44:29.469028 18467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "ac634b3bed1740af968cd1ca61c511fd"
format_stamp: "Formatted at 2026-09-26 16:44:29 on dist-test-slave-wt2j"
server_key: "1605550bb4d267bd2d27e46d558c5cae"
server_key_iv: "23e20522db46fe437e5088428c0ca594"
server_key_version: "encryptionkey@0"
I20260926 16:44:29.469316 18467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:29.491338 18467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:29.496541 18467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:29.496881 18467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:29.521224 18467 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:35741
I20260926 16:44:29.521281 18533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:35741 every 8 connection(s)
I20260926 16:44:29.523001 18467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:44:29.527450 18534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:29.529714 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 18467
I20260926 16:44:29.530002 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:44:29.530678 10171 external_mini_cluster.cc:1428] Setting key 3c2f7f219ef84d97070dce477fa67684
I20260926 16:44:29.540870 18534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd: Bootstrap starting.
I20260926 16:44:29.544998 18534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:29.546419 18534 log.cc:824] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd: Log is configured to *not* fsync() on all Append() calls
Sep 26 16:44:29 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:44:29.550186 18534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd: No bootstrap required, opened a new log
I20260926 16:44:29.558673 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:29.533210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54196 (local address 127.9.238.254:35741)
0926 16:44:29.534184 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:29.534198 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:29.534252 (+ 54us) server_negotiation.cc:407] Connection header received
0926 16:44:29.536333 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:29.536361 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:29.536896 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:29.537327 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:29.539331 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:29.540945 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:29.542852 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:29.543836 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:29.544010 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:29.549064 (+ 5054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:29.549074 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:29.549095 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:29.549142 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:29.553057 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:29.554029 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:29.554037 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:29.554045 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:29.554262 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:29.554722 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:29.554727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:29.554729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:29.555738 (+ 1009us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:29.556043 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:29.556643 (+ 600us) server_negotiation.cc:299] Negotiation successful
0926 16:44:29.557239 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":461,"threads_started":1}
I20260926 16:44:29.559680 18534 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac634b3bed1740af968cd1ca61c511fd" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35741 } }
I20260926 16:44:29.560155 18534 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:29.560267 18534 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac634b3bed1740af968cd1ca61c511fd, State: Initialized, Role: FOLLOWER
I20260926 16:44:29.560880 18534 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac634b3bed1740af968cd1ca61c511fd" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35741 } }
I20260926 16:44:29.561133 18534 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:44:29.561285 18534 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:44:29.561821 18534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:29.564628 18534 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac634b3bed1740af968cd1ca61c511fd" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35741 } }
I20260926 16:44:29.565158 18534 leader_election.cc:302] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac634b3bed1740af968cd1ca61c511fd; no voters:
I20260926 16:44:29.565871 18534 leader_election.cc:288] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:44:29.566128 18539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:44:29.567495 18539 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [term 1 LEADER]: Becoming Leader. State: Replica: ac634b3bed1740af968cd1ca61c511fd, State: Running, Role: LEADER
I20260926 16:44:29.568231 18539 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac634b3bed1740af968cd1ca61c511fd" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35741 } }
I20260926 16:44:29.568372 18534 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:44:29.573612 18540 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac634b3bed1740af968cd1ca61c511fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac634b3bed1740af968cd1ca61c511fd" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35741 } } }
I20260926 16:44:29.574189 18540 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:29.574712 18541 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac634b3bed1740af968cd1ca61c511fd. Latest consensus state: current_term: 1 leader_uuid: "ac634b3bed1740af968cd1ca61c511fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac634b3bed1740af968cd1ca61c511fd" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35741 } } }
I20260926 16:44:29.575031 18541 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:29.576596 18552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:44:29.582912 18552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:44:29.593492 18552 catalog_manager.cc:1347] Generated new cluster ID: 8633fe08e6084903b6c7a7d9606367aa
I20260926 16:44:29.593657 18552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:44:29.612911 18552 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:44:29.615039 18552 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:44:29.636579 18552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd: Generated new TSK 0
I20260926 16:44:29.637799 18552 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
I20260926 16:44:29.712888 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:0
--local_ip_for_outbound_sockets=127.9.238.193
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:35741
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:37971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:30.132094 18560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:30.132516 18560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:30.132632 18560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:30.141278 18560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:30.141436 18560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:30.141680 18560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:44:30.153425 18560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:37971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:35741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:30.155238 18560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:30.156996 18560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:30.173996 18568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:30.173980 18565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:30.173935 18566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:30.182335 18560 server_base.cc:1034] running on GCE node
I20260926 16:44:30.183393 18560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:30.184823 18560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:30.186097 18560 hybrid_clock.cc:648] HybridClock initialized: now 1790441070186034 us; error 54 us; skew 500 ppm
I20260926 16:44:30.186498 18560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:30 dist-test-slave-wt2j krb5kdc[18454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441070, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:30.198020 18560 init.cc:377] Logged in from keytab as kudu/127.9.238.193@KRBTEST.COM (short username kudu)
I20260926 16:44:30.200296 18560 webserver.cc:460] Webserver started at http://127.9.238.193:38669/ using document root <none> and password file <none>
I20260926 16:44:30.201313 18560 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:30.201478 18560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:30.201882 18560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:30.205753 18560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/data/instance:
uuid: "e61731caa3954b83b59a4e3347208098"
format_stamp: "Formatted at 2026-09-26 16:44:30 on dist-test-slave-wt2j"
server_key: "305d8a82f3ab1aece4da8d4d5adc14bd"
server_key_iv: "520bdeab22b2b2f72130482ef69a7736"
server_key_version: "encryptionkey@0"
I20260926 16:44:30.206789 18560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance:
uuid: "e61731caa3954b83b59a4e3347208098"
format_stamp: "Formatted at 2026-09-26 16:44:30 on dist-test-slave-wt2j"
server_key: "305d8a82f3ab1aece4da8d4d5adc14bd"
server_key_iv: "520bdeab22b2b2f72130482ef69a7736"
server_key_version: "encryptionkey@0"
I20260926 16:44:30.213698 18560 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.007s
I20260926 16:44:30.218617 18575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:30.220901 18560 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 16:44:30.221201 18560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "e61731caa3954b83b59a4e3347208098"
format_stamp: "Formatted at 2026-09-26 16:44:30 on dist-test-slave-wt2j"
server_key: "305d8a82f3ab1aece4da8d4d5adc14bd"
server_key_iv: "520bdeab22b2b2f72130482ef69a7736"
server_key_version: "encryptionkey@0"
I20260926 16:44:30.221508 18560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:30.244313 18560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:30.250274 18560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:30.250660 18560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:30.252038 18560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:30.254118 18560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:30.254240 18560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:30.254487 18560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:30.254618 18560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:30.296945 18560 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:34485
I20260926 16:44:30.296993 18687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:34485 every 8 connection(s)
I20260926 16:44:30.298561 18560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:44:30.303642 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 18560
I20260926 16:44:30.303901 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance
I20260926 16:44:30.304764 10171 external_mini_cluster.cc:1428] Setting key 1a77a0a8d98130c6cef0a76770f63e97
Sep 26 16:44:30 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441070, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:44:30.323997 18690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:30.301690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:50945 (local address 127.9.238.254:35741)
0926 16:44:30.302240 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:30.302244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:30.303321 (+ 1077us) server_negotiation.cc:407] Connection header received
0926 16:44:30.305701 (+ 2380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:30.305707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:30.305804 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:30.305961 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:30.307910 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:30.309173 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:30.311079 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:30.311913 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:30.312094 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:30.316102 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:30.316113 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:30.316123 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:30.316189 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:30.319717 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:30.320924 (+ 1207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:30.320929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:30.320932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:30.321092 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:30.321663 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:30.321667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:30.321670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:30.322015 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:30.322163 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:30.323279 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0926 16:44:30.323543 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":339,"threads_started":1}
I20260926 16:44:30.325351 18691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:30.302227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.193:50945)
0926 16:44:30.303128 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:30.303178 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:30.305302 (+ 2124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:30.306276 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:30.306286 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:30.306975 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:30.307711 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:30.307731 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:30.309416 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:30.309420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:30.310877 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:30.310891 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:30.312206 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:30.312212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:30.312443 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:30.313308 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:30.313337 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:30.315753 (+ 2416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:30.319950 (+ 4197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:30.319964 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:30.319989 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:30.320729 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:30.321220 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:30.321227 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:30.321234 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:30.321504 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:30.322158 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:30.322170 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:30.322706 (+ 536us) client_negotiation.cc:769] Sending connection context
0926 16:44:30.323060 (+ 354us) client_negotiation.cc:240] Negotiation successful
0926 16:44:30.323751 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":321,"threads_started":1}
I20260926 16:44:30.327518 18688 heartbeater.cc:347] Connected to a master server at 127.9.238.254:35741
I20260926 16:44:30.328038 18688 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:30.329188 18688 heartbeater.cc:511] Master 127.9.238.254:35741 requested a full tablet report, sending...
I20260926 16:44:30.332293 18499 ts_manager.cc:194] Registered new tserver with Master: e61731caa3954b83b59a4e3347208098 (127.9.238.193:34485)
WARNING: no policy specified for kudu/127.9.238.194@KRBTEST.COM; defaulting to no policy
I20260926 16:44:30.334208 18499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.193@KRBTEST.COM'} at 127.9.238.193:50945
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
I20260926 16:44:30.366806 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:0
--local_ip_for_outbound_sockets=127.9.238.194
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:35741
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:37971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:30.735176 18694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:30.735476 18694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:30.735659 18694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:30.743124 18694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:30.743309 18694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:30.743566 18694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:44:30.754094 18694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:37971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:35741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:30.755439 18694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:30.756879 18694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:30.772214 18702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:30.772317 18699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:30.773044 18700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:30.773516 18694 server_base.cc:1034] running on GCE node
I20260926 16:44:30.774358 18694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:30.775521 18694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:30.776829 18694 hybrid_clock.cc:648] HybridClock initialized: now 1790441070776738 us; error 75 us; skew 500 ppm
I20260926 16:44:30.777288 18694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:30 dist-test-slave-wt2j krb5kdc[18454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441070, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:30.780938 18694 init.cc:377] Logged in from keytab as kudu/127.9.238.194@KRBTEST.COM (short username kudu)
I20260926 16:44:30.782819 18694 webserver.cc:460] Webserver started at http://127.9.238.194:44943/ using document root <none> and password file <none>
I20260926 16:44:30.783646 18694 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:30.783763 18694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:30.784126 18694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:30.787156 18694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/data/instance:
uuid: "66703f10c9e841abbb74f21b273b9c07"
format_stamp: "Formatted at 2026-09-26 16:44:30 on dist-test-slave-wt2j"
server_key: "e48d4f74fc39cfa06e20d1c9d2d10d84"
server_key_iv: "f2fe5bbfb62fceecb2397beab5c12f7f"
server_key_version: "encryptionkey@0"
I20260926 16:44:30.788018 18694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance:
uuid: "66703f10c9e841abbb74f21b273b9c07"
format_stamp: "Formatted at 2026-09-26 16:44:30 on dist-test-slave-wt2j"
server_key: "e48d4f74fc39cfa06e20d1c9d2d10d84"
server_key_iv: "f2fe5bbfb62fceecb2397beab5c12f7f"
server_key_version: "encryptionkey@0"
I20260926 16:44:30.793921 18694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:44:30.798297 18709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:30.800186 18694 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 16:44:30.800408 18694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "66703f10c9e841abbb74f21b273b9c07"
format_stamp: "Formatted at 2026-09-26 16:44:30 on dist-test-slave-wt2j"
server_key: "e48d4f74fc39cfa06e20d1c9d2d10d84"
server_key_iv: "f2fe5bbfb62fceecb2397beab5c12f7f"
server_key_version: "encryptionkey@0"
I20260926 16:44:30.800649 18694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:30.814289 18694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:30.819607 18694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:30.819972 18694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:30.821206 18694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:30.822880 18694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:30.822992 18694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:30.823273 18694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:30.823361 18694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:30.862058 18694 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:35017
I20260926 16:44:30.862103 18821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:35017 every 8 connection(s)
I20260926 16:44:30.863373 18694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:44:30.874003 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 18694
I20260926 16:44:30.874254 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance
I20260926 16:44:30.874818 10171 external_mini_cluster.cc:1428] Setting key cea7655ed613e58a440afbe3f8fb27ae
Sep 26 16:44:30 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441070, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:44:30.890870 18824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:30.866780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:42755 (local address 127.9.238.254:35741)
0926 16:44:30.867307 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:30.867312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:30.868684 (+ 1372us) server_negotiation.cc:407] Connection header received
0926 16:44:30.871364 (+ 2680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:30.871371 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:30.871496 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:30.871714 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:30.874279 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:30.875335 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:30.878094 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:30.878544 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:30.878667 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:30.884147 (+ 5480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:30.884152 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:30.884155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:30.884195 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:30.886905 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:30.887839 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:30.887843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:30.887846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:30.887965 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:30.888420 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:30.888427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:30.888433 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:30.888838 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:30.889016 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:30.890066 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0926 16:44:30.890357 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":242,"threads_started":1}
I20260926 16:44:30.892103 18825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:30.867351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.194:42755)
0926 16:44:30.868455 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:30.868530 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:30.870957 (+ 2427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:30.872041 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:30.872053 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:30.872804 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:30.873880 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:30.873922 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:30.875570 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:30.875579 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:30.877864 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:30.877892 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:30.878851 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:30.878861 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:30.879308 (+ 447us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:30.880375 (+ 1067us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:30.880417 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:30.883859 (+ 3442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:30.887101 (+ 3242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:30.887112 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:30.887135 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:30.887704 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:30.888083 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:30.888090 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:30.888094 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:30.888299 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:30.889039 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:30.889050 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:30.889572 (+ 522us) client_negotiation.cc:769] Sending connection context
0926 16:44:30.889903 (+ 331us) client_negotiation.cc:240] Negotiation successful
0926 16:44:30.890461 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":388,"threads_started":1}
I20260926 16:44:30.893873 18822 heartbeater.cc:347] Connected to a master server at 127.9.238.254:35741
I20260926 16:44:30.894243 18822 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:30.895112 18822 heartbeater.cc:511] Master 127.9.238.254:35741 requested a full tablet report, sending...
I20260926 16:44:30.896867 18499 ts_manager.cc:194] Registered new tserver with Master: 66703f10c9e841abbb74f21b273b9c07 (127.9.238.194:35017)
I20260926 16:44:30.897804 18499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.194@KRBTEST.COM'} at 127.9.238.194:42755
WARNING: no policy specified for kudu/127.9.238.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
I20260926 16:44:30.935539 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:0
--local_ip_for_outbound_sockets=127.9.238.195
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:35741
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:37971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:31.277136 18828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:31.277432 18828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:31.277529 18828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:31.286140 18828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:31.286319 18828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:31.286540 18828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:44:31.299821 18828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:37971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:35741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:31.301440 18828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:31.303195 18828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:31.318092 18834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:31.318086 18836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:31.318342 18833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:31.320279 18828 server_base.cc:1034] running on GCE node
I20260926 16:44:31.321126 18828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:31.322436 18828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:31.323673 18828 hybrid_clock.cc:648] HybridClock initialized: now 1790441071323620 us; error 45 us; skew 500 ppm
I20260926 16:44:31.324095 18828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:31 dist-test-slave-wt2j krb5kdc[18454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441071, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:31.328465 18828 init.cc:377] Logged in from keytab as kudu/127.9.238.195@KRBTEST.COM (short username kudu)
I20260926 16:44:31.330269 18828 webserver.cc:460] Webserver started at http://127.9.238.195:36467/ using document root <none> and password file <none>
I20260926 16:44:31.331017 18828 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:31.331126 18828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:31.331403 18828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:31.334612 18828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/data/instance:
uuid: "4b52dcf3801d4f8a8b1ff5f474566eca"
format_stamp: "Formatted at 2026-09-26 16:44:31 on dist-test-slave-wt2j"
server_key: "be0b67956b0787d58eb09d4d89164c36"
server_key_iv: "2cc8505f8b79276e219506eda5eaf6bb"
server_key_version: "encryptionkey@0"
I20260926 16:44:31.335446 18828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance:
uuid: "4b52dcf3801d4f8a8b1ff5f474566eca"
format_stamp: "Formatted at 2026-09-26 16:44:31 on dist-test-slave-wt2j"
server_key: "be0b67956b0787d58eb09d4d89164c36"
server_key_iv: "2cc8505f8b79276e219506eda5eaf6bb"
server_key_version: "encryptionkey@0"
I20260926 16:44:31.338580 18688 heartbeater.cc:503] Master 127.9.238.254:35741 was elected leader, sending a full tablet report...
I20260926 16:44:31.341755 18828 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260926 16:44:31.345980 18843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:31.347765 18828 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:44:31.347971 18828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "4b52dcf3801d4f8a8b1ff5f474566eca"
format_stamp: "Formatted at 2026-09-26 16:44:31 on dist-test-slave-wt2j"
server_key: "be0b67956b0787d58eb09d4d89164c36"
server_key_iv: "2cc8505f8b79276e219506eda5eaf6bb"
server_key_version: "encryptionkey@0"
I20260926 16:44:31.348194 18828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:31.372339 18828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:31.377820 18828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:31.378191 18828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:31.379405 18828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:31.381032 18828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:31.381117 18828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:31.381255 18828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:31.381309 18828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:31.421832 18828 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:42115
I20260926 16:44:31.421851 18955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:42115 every 8 connection(s)
I20260926 16:44:31.423724 18828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:44:31.430372 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 18828
I20260926 16:44:31.430564 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance
I20260926 16:44:31.431083 10171 external_mini_cluster.cc:1428] Setting key 94214dbf412dadffa49ab767a33c661c
Sep 26 16:44:31 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441071, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:44:31.449738 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:31.427024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:55711 (local address 127.9.238.254:35741)
0926 16:44:31.427568 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:31.427573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:31.429022 (+ 1449us) server_negotiation.cc:407] Connection header received
0926 16:44:31.431347 (+ 2325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:31.431355 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:31.431494 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:31.431716 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:31.433812 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:31.434943 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:31.436903 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:31.437441 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:31.437582 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:31.441714 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:31.441720 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:31.441725 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:31.441771 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:31.445076 (+ 3305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:31.446322 (+ 1246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:31.446331 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:31.446338 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:31.446538 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:31.447253 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:31.447262 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:31.447269 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:31.447833 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:31.447990 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:31.449108 (+ 1118us) server_negotiation.cc:299] Negotiation successful
0926 16:44:31.449401 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":311,"threads_started":1}
I20260926 16:44:31.450794 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:31.427768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.195:55711)
0926 16:44:31.428751 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:31.428802 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:31.430919 (+ 2117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:31.432112 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:31.432128 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:31.432863 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:31.433591 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:31.433624 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:31.435129 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:31.435134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:31.436691 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:31.436712 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:31.437762 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:31.437769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:31.438035 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:31.438958 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:31.439004 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:31.441391 (+ 2387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:31.445311 (+ 3920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:31.445324 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:31.445355 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:31.446132 (+ 777us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:31.446686 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:31.446693 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:31.446697 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:31.446991 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:31.448119 (+ 1128us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:31.448125 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:31.448628 (+ 503us) client_negotiation.cc:769] Sending connection context
0926 16:44:31.448958 (+ 330us) client_negotiation.cc:240] Negotiation successful
0926 16:44:31.449404 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":393,"threads_started":1}
I20260926 16:44:31.452997 18956 heartbeater.cc:347] Connected to a master server at 127.9.238.254:35741
I20260926 16:44:31.453367 18956 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:31.454135 18956 heartbeater.cc:511] Master 127.9.238.254:35741 requested a full tablet report, sending...
I20260926 16:44:31.455847 18499 ts_manager.cc:194] Registered new tserver with Master: 4b52dcf3801d4f8a8b1ff5f474566eca (127.9.238.195:42115)
I20260926 16:44:31.456728 18499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.195@KRBTEST.COM'} at 127.9.238.195:55711
I20260926 16:44:31.461580 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.9.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.193.keytab.
Entry for principal oryx/127.9.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.193.keytab.
WARNING: no policy specified for oryx/127.9.238.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.194.keytab.
Entry for principal oryx/127.9.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.194.keytab.
WARNING: no policy specified for oryx/127.9.238.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.9.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.9.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.195.keytab.
Entry for principal oryx/127.9.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.195.keytab.
I20260926 16:44:31.639318 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 18560
Sep 26 16:44:31 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.193@KRBTEST.COM
I20260926 16:44:31.659909 18967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:31.640996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39894 (local address 127.9.238.193:34485)
0926 16:44:31.641923 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:31.641932 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:31.641968 (+ 36us) server_negotiation.cc:407] Connection header received
0926 16:44:31.642053 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:31.642061 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:31.642360 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:31.642550 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:31.644097 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:31.645334 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:31.646738 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:31.647446 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:31.647684 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:31.652330 (+ 4646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:31.652350 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:31.652372 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:31.652435 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:31.656225 (+ 3790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:31.657139 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:31.657149 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:31.657152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:31.657424 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:31.657957 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:31.657962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:31.657964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:31.658437 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:31.658734 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:31.659094 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 16:44:31.659454 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":538,"threads_started":1}
I20260926 16:44:31.661580 18683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:31.751070 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 18560
I20260926 16:44:31.768539 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 18694
Sep 26 16:44:31 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.194@KRBTEST.COM
I20260926 16:44:31.787467 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:31.769587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38340 (local address 127.9.238.194:35017)
0926 16:44:31.770233 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:31.770244 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:31.770283 (+ 39us) server_negotiation.cc:407] Connection header received
0926 16:44:31.770352 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:31.770359 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:31.770582 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:31.770743 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:31.772330 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:31.773444 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:31.774959 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:31.775803 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:31.775932 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:31.781164 (+ 5232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:31.781180 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:31.781198 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:31.781240 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:31.784428 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:31.785378 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:31.785386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:31.785388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:31.785514 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:31.785996 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:31.786000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:31.786002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:31.786395 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:31.786658 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:31.786965 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 16:44:31.787171 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":352,"threads_started":1}
I20260926 16:44:31.789377 18817 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:31.880065 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 18694
I20260926 16:44:31.896793 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 18828
Sep 26 16:44:31 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.195@KRBTEST.COM
I20260926 16:44:31.916368 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:31.897784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60854 (local address 127.9.238.195:42115)
0926 16:44:31.898375 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:31.898382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:31.898417 (+ 35us) server_negotiation.cc:407] Connection header received
0926 16:44:31.898486 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:31.898492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:31.898717 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:31.898882 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:31.900688 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:31.901796 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:31.903337 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:31.904041 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:31.904235 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:31.909772 (+ 5537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:31.909789 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:31.909806 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:31.909857 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:31.913359 (+ 3502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:31.914182 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:31.914190 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:31.914192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:31.914327 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:31.914712 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:31.914716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:31.914719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:31.915097 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:31.915342 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:31.915823 (+ 481us) server_negotiation.cc:299] Negotiation successful
0926 16:44:31.916049 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":287,"threads_started":1}
I20260926 16:44:31.917717 18951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:32.015940 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 18828
I20260926 16:44:32.034365 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 18467
I20260926 16:44:32.035868 18529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:32.143653 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 18467
I20260926 16:44:32.169039 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:35741
--webserver_interface=127.9.238.254
--webserver_port=35863
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:37971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:35741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:32.546561 18974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:32.546891 18974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:32.546983 18974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:32.554766 18974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:44:32.554901 18974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:32.554950 18974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:32.555004 18974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:44:32.555065 18974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:44:32.565735 18974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:37971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:35741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:35741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=35863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:32.567256 18974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:32.568650 18974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:32.582119 18982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:32.582312 18980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:32.582588 18979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:32.583324 18974 server_base.cc:1034] running on GCE node
I20260926 16:44:32.584266 18974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:32.585857 18974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:32.587199 18974 hybrid_clock.cc:648] HybridClock initialized: now 1790441072587092 us; error 110 us; skew 500 ppm
I20260926 16:44:32.587688 18974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:32 dist-test-slave-wt2j krb5kdc[18454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441072, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:32.591045 18974 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:44:32.592760 18974 webserver.cc:460] Webserver started at http://127.9.238.254:35863/ using document root <none> and password file <none>
I20260926 16:44:32.593495 18974 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:32.593618 18974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:32.600059 18974 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.002s
I20260926 16:44:32.603508 18989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:32.605362 18974 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:44:32.605589 18974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "ac634b3bed1740af968cd1ca61c511fd"
format_stamp: "Formatted at 2026-09-26 16:44:29 on dist-test-slave-wt2j"
server_key: "1605550bb4d267bd2d27e46d558c5cae"
server_key_iv: "23e20522db46fe437e5088428c0ca594"
server_key_version: "encryptionkey@0"
I20260926 16:44:32.606715 18974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:32.654736 18974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:32.660265 18974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:32.660574 18974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:32.682271 18974 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:35741
I20260926 16:44:32.682315 19040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:35741 every 8 connection(s)
I20260926 16:44:32.683892 18974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:44:32.689791 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 18974
I20260926 16:44:32.691001 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:34485
--local_ip_for_outbound_sockets=127.9.238.193
--tserver_master_addrs=127.9.238.254:35741
--webserver_port=38669
--webserver_interface=127.9.238.193
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:37971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.238.193
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:44:32.694744 19041 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:44:32.701771 19041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd: Bootstrap starting.
I20260926 16:44:32.726526 19041 log.cc:824] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:32.738247 19041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:44:32.738968 19041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd: Bootstrap complete.
I20260926 16:44:32.749658 19041 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac634b3bed1740af968cd1ca61c511fd" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35741 } }
I20260926 16:44:32.750674 19041 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac634b3bed1740af968cd1ca61c511fd, State: Initialized, Role: FOLLOWER
I20260926 16:44:32.751650 19041 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac634b3bed1740af968cd1ca61c511fd" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35741 } }
I20260926 16:44:32.751948 19041 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:44:32.752085 19041 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:44:32.752306 19041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [term 1 FOLLOWER]: Advancing to term 2
I20260926 16:44:32.757251 19041 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac634b3bed1740af968cd1ca61c511fd" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35741 } }
I20260926 16:44:32.758067 19041 leader_election.cc:302] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac634b3bed1740af968cd1ca61c511fd; no voters:
I20260926 16:44:32.758842 19041 leader_election.cc:288] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 16:44:32.759035 19045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [term 2 FOLLOWER]: Leader election won for term 2
I20260926 16:44:32.760532 19045 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [term 2 LEADER]: Becoming Leader. State: Replica: ac634b3bed1740af968cd1ca61c511fd, State: Running, Role: LEADER
I20260926 16:44:32.761350 19045 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac634b3bed1740af968cd1ca61c511fd" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35741 } }
I20260926 16:44:32.761824 19041 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:44:32.764518 19047 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac634b3bed1740af968cd1ca61c511fd. Latest consensus state: current_term: 2 leader_uuid: "ac634b3bed1740af968cd1ca61c511fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac634b3bed1740af968cd1ca61c511fd" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35741 } } }
I20260926 16:44:32.764979 19047 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:32.765518 19046 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ac634b3bed1740af968cd1ca61c511fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac634b3bed1740af968cd1ca61c511fd" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35741 } } }
I20260926 16:44:32.766233 19046 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:32.766580 19051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:44:32.787178 19051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:44:32.789686 19051 catalog_manager.cc:1259] Loaded cluster ID: 8633fe08e6084903b6c7a7d9606367aa
I20260926 16:44:32.790019 19051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:44:32.796547 19051 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:44:32.799057 19051 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ac634b3bed1740af968cd1ca61c511fd: Loaded TSK: 0
I20260926 16:44:32.801746 19051 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 16:44:33.084355 19043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:33.084662 19043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:33.084786 19043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:33.092501 19043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:33.092634 19043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:33.092803 19043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:44:33.103849 19043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:37971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.193.keytab
--principal=oryx/127.9.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:34485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=38669
--tserver_master_addrs=127.9.238.254:35741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:33.105422 19043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:33.106796 19043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:33.122272 19067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:33.122603 19068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:33.123208 19070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:33.123723 19043 server_base.cc:1034] running on GCE node
I20260926 16:44:33.124409 19043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:33.125497 19043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:33.128834 19043 hybrid_clock.cc:648] HybridClock initialized: now 1790441073128728 us; error 98 us; skew 500 ppm
I20260926 16:44:33.129251 19043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:33 dist-test-slave-wt2j krb5kdc[18454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441073, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:33.132870 19043 init.cc:377] Logged in from keytab as oryx/127.9.238.193@KRBTEST.COM (short username oryx)
I20260926 16:44:33.134651 19043 webserver.cc:460] Webserver started at http://127.9.238.193:38669/ using document root <none> and password file <none>
I20260926 16:44:33.135399 19043 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:33.135542 19043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:33.141227 19043 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 16:44:33.144654 19077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:33.146646 19043 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 16:44:33.146854 19043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "e61731caa3954b83b59a4e3347208098"
format_stamp: "Formatted at 2026-09-26 16:44:30 on dist-test-slave-wt2j"
server_key: "305d8a82f3ab1aece4da8d4d5adc14bd"
server_key_iv: "520bdeab22b2b2f72130482ef69a7736"
server_key_version: "encryptionkey@0"
I20260926 16:44:33.147948 19043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:33.166393 19043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:33.171072 19043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:33.171353 19043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:33.172533 19043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:33.174015 19043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:33.174144 19043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:33.174275 19043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:33.174365 19043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:33.217797 19043 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:34485
I20260926 16:44:33.217989 19189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:34485 every 8 connection(s)
I20260926 16:44:33.219481 19043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:44:33.226042 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 19043
I20260926 16:44:33.226846 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:35017
--local_ip_for_outbound_sockets=127.9.238.194
--tserver_master_addrs=127.9.238.254:35741
--webserver_port=44943
--webserver_interface=127.9.238.194
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:37971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.238.194
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:44:33 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:33 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 16:44:33.586906 19194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:33.587276 19194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:33.587440 19194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:33.595244 19194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:33.595412 19194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:33.595642 19194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:44:33.606128 19194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:37971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.194.keytab
--principal=oryx/127.9.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:35017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=44943
--tserver_master_addrs=127.9.238.254:35741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:33.607625 19194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:33.608975 19194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:33.625041 19203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:33.624908 19200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:33.625914 19201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:33.627775 19194 server_base.cc:1034] running on GCE node
I20260926 16:44:33.628545 19194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:33.629766 19194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:33.633159 19194 hybrid_clock.cc:648] HybridClock initialized: now 1790441073633018 us; error 128 us; skew 500 ppm
I20260926 16:44:33.633559 19194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:33 dist-test-slave-wt2j krb5kdc[18454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441073, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:33.637446 19194 init.cc:377] Logged in from keytab as oryx/127.9.238.194@KRBTEST.COM (short username oryx)
I20260926 16:44:33.639482 19194 webserver.cc:460] Webserver started at http://127.9.238.194:44943/ using document root <none> and password file <none>
I20260926 16:44:33.640316 19194 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:33.640478 19194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:33.646863 19194 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 16:44:33.650725 19210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:33.653483 19194 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260926 16:44:33.653730 19194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "66703f10c9e841abbb74f21b273b9c07"
format_stamp: "Formatted at 2026-09-26 16:44:30 on dist-test-slave-wt2j"
server_key: "e48d4f74fc39cfa06e20d1c9d2d10d84"
server_key_iv: "f2fe5bbfb62fceecb2397beab5c12f7f"
server_key_version: "encryptionkey@0"
I20260926 16:44:33.654788 19194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:33.689079 19194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:33.693761 19194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:33.694120 19194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:33.695241 19194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:33.696777 19194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:33.696902 19194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:33.697041 19194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:33.697142 19194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:33.748473 19194 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:35017
I20260926 16:44:33.748530 19322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:35017 every 8 connection(s)
I20260926 16:44:33.749994 19194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:44:33.755667 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 19194
I20260926 16:44:33.756582 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:42115
--local_ip_for_outbound_sockets=127.9.238.195
--tserver_master_addrs=127.9.238.254:35741
--webserver_port=36467
--webserver_interface=127.9.238.195
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:37971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.9.238.195
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:44:33 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:33 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 16:44:34.354787 19327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:34.355579 19327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:34.355830 19327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:34.377703 19327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:34.378023 19327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:34.378398 19327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:44:34.407274 19327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:37971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/krb5kdc/oryx_127.9.238.195.keytab
--principal=oryx/127.9.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:42115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=36467
--tserver_master_addrs=127.9.238.254:35741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:34.410504 19327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:34.414363 19327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:34.446448 19333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:34.447993 19334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:34.449548 19336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:34.449846 19327 server_base.cc:1034] running on GCE node
I20260926 16:44:34.450804 19327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:34.452435 19327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:34.455814 19327 hybrid_clock.cc:648] HybridClock initialized: now 1790441074455688 us; error 132 us; skew 500 ppm
I20260926 16:44:34.456295 19327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:34 dist-test-slave-wt2j krb5kdc[18454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441074, etypes {rep=17 tkt=17 ses=17}, oryx/127.9.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:34.461623 19327 init.cc:377] Logged in from keytab as oryx/127.9.238.195@KRBTEST.COM (short username oryx)
I20260926 16:44:34.464946 19327 webserver.cc:460] Webserver started at http://127.9.238.195:36467/ using document root <none> and password file <none>
I20260926 16:44:34.466037 19327 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:34.466195 19327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:34.475729 19327 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.006s sys 0.000s
I20260926 16:44:34.480538 19343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:34.483018 19327 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260926 16:44:34.483251 19327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "4b52dcf3801d4f8a8b1ff5f474566eca"
format_stamp: "Formatted at 2026-09-26 16:44:31 on dist-test-slave-wt2j"
server_key: "be0b67956b0787d58eb09d4d89164c36"
server_key_iv: "2cc8505f8b79276e219506eda5eaf6bb"
server_key_version: "encryptionkey@0"
I20260926 16:44:34.484733 19327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:34.506619 19327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:34.512825 19327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:34.513144 19327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:34.514652 19327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
W20260926 16:44:34.515030 19192 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.270s user 0.006s sys 0.014s
I20260926 16:44:34.517414 19192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:33.223826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.193:50991)
0926 16:44:33.224737 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:33.224788 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:33.227329 (+ 2541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:33.234162 (+ 6833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:33.234175 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:33.234823 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:33.235502 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:33.235529 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:33.239027 (+ 3498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:33.239037 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:33.241396 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:33.241420 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:33.243428 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:33.243439 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:33.243865 (+ 426us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:33.244911 (+ 1046us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:33.244954 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:34.515966 (+1271012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":378,"threads_started":1}
W20260926 16:44:34.519538 19190 heartbeater.cc:644] Failed to heartbeat to 127.9.238.254:35741 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.238.254:35741: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
W20260926 16:44:34.520311 19193 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:33.223923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:50991 (local address 127.9.238.254:35741)
0926 16:44:33.232475 (+ 8552us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:33.232497 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:33.232564 (+ 67us) server_negotiation.cc:407] Connection header received
0926 16:44:33.233004 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:33.233028 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:33.233569 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:33.233998 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:33.235887 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:33.238693 (+ 2806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:33.241673 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:33.243042 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:33.243346 (+ 304us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:34.519076 (+1275730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.193:50991: BlockingRecv error: recv got EOF from 127.9.238.193:50991 (error 108)
Metrics: {"server-negotiator.queue_time_us":8171,"thread_start_us":7918,"threads_started":1}
I20260926 16:44:34.524610 19327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:34.524748 19327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:34.524863 19327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:34.524930 19327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 16:44:34 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:34 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 16:44:34.540877 19193 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:34.522192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:36429 (local address 127.9.238.254:35741)
0926 16:44:34.522437 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:34.522446 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:34.522478 (+ 32us) server_negotiation.cc:407] Connection header received
0926 16:44:34.522604 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:34.522613 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:34.522767 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:34.522970 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:34.524551 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:34.525893 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:34.529193 (+ 3300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:34.530251 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:34.530433 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:34.540428 (+ 9995us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.193:36429: BlockingRecv error: recv got EOF from 127.9.238.193:36429 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 16:44:34.541635 19192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:34.521625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.193:36429)
0926 16:44:34.521995 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:34.522038 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:34.522298 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:34.523058 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:34.523067 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:34.523919 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:34.524296 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:34.524316 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:34.526162 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:34.526174 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:34.528846 (+ 2672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:34.528876 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:34.530958 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:34.530969 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:34.531298 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:34.532511 (+ 1213us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:34.532554 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:34.541196 (+ 8642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112}
Sep 26 16:44:34 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:34 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 16:44:34.562631 19193 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:34.543760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:58577 (local address 127.9.238.254:35741)
0926 16:44:34.544402 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:34.544412 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:34.544439 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:44:34.544539 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:34.544548 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:34.544688 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:34.544824 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:34.546588 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:34.548489 (+ 1901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:34.551610 (+ 3121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:34.552693 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:34.552885 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:34.562125 (+ 9240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.193:58577: BlockingRecv error: recv got EOF from 127.9.238.193:58577 (error 108)
Metrics: {"server-negotiator.queue_time_us":440}
I20260926 16:44:34.562687 19192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:34.542874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.193:58577)
0926 16:44:34.543498 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:34.543531 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:34.544050 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:34.545122 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:34.545132 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:34.545877 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:34.546287 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:34.546306 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:34.548791 (+ 2485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:34.548800 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:34.551260 (+ 2460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:34.551288 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:34.553274 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:34.553285 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:34.553630 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:34.554855 (+ 1225us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:34.554894 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:34.562215 (+ 7321us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393}
W20260926 16:44:34.563318 19190 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 16:44:34.582491 19327 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:42115
I20260926 16:44:34.582531 19455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:42115 every 8 connection(s)
I20260926 16:44:34.584158 19327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:44:34.585790 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 19327
Sep 26 16:44:34 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:34 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:34.626392 19193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:34.587699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43816 (local address 127.9.238.254:35741)
0926 16:44:34.587979 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:34.587990 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:34.601870 (+ 13880us) server_negotiation.cc:407] Connection header received
0926 16:44:34.602255 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:34.602267 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:34.602440 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:34.602666 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:34.604733 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:34.606586 (+ 1853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:34.608454 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:34.609167 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:34.609305 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:34.615371 (+ 6066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:34.615386 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:34.615416 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:34.615486 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:34.620401 (+ 4915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:34.622034 (+ 1633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:34.622057 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:34.622070 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:34.622337 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:34.622966 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:34.622974 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:34.622980 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:34.623832 (+ 852us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:34.624163 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:34.625120 (+ 957us) server_negotiation.cc:299] Negotiation successful
0926 16:44:34.625790 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 16:44:35.153097 19326 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.357s user 0.001s sys 0.027s
W20260926 16:44:35.155128 19325 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:33.755000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:60305 (local address 127.9.238.254:35741)
0926 16:44:33.768146 (+ 13146us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:33.768155 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:33.770566 (+ 2411us) server_negotiation.cc:407] Connection header received
0926 16:44:33.774403 (+ 3837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:33.774413 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:33.774564 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:33.774709 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:33.779372 (+ 4663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:33.780865 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:33.785718 (+ 4853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:33.786639 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:33.786838 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:35.154832 (+1367994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.194:60305: BlockingRecv error: recv got EOF from 127.9.238.194:60305 (error 108)
Metrics: {"server-negotiator.queue_time_us":12926,"thread_start_us":12728,"threads_started":1}
I20260926 16:44:35.156399 19326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:33.755521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.194:60305)
0926 16:44:33.768917 (+ 13396us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:33.769037 (+ 120us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:33.773684 (+ 4647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:33.775288 (+ 1604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:33.775314 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:33.777102 (+ 1788us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:33.778897 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:33.778973 (+ 76us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:33.781230 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:33.781241 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:33.785334 (+ 4093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:33.785381 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:33.787300 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:33.787326 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:33.788035 (+ 709us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:33.796305 (+ 8270us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:33.796365 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:35.154524 (+1358159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":12686,"thread_start_us":12392,"threads_started":1}
W20260926 16:44:35.157909 19323 heartbeater.cc:644] Failed to heartbeat to 127.9.238.254:35741 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.238.254:35741: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Sep 26 16:44:35 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:35 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:35.171622 19326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:35.158948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.194:54101)
0926 16:44:35.159214 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:35.159236 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:35.159416 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:35.159906 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:35.159911 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:35.160533 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:35.160797 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:35.160808 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:35.162167 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:35.162174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:35.163799 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:35.163817 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:35.164711 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:35.164718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:35.164938 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:35.165790 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:35.165812 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:35.171112 (+ 5300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":86}
W20260926 16:44:35.171824 19325 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:35.159199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:54101 (local address 127.9.238.254:35741)
0926 16:44:35.159469 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:35.159479 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:35.159507 (+ 28us) server_negotiation.cc:407] Connection header received
0926 16:44:35.159623 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:35.159629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:35.159744 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:35.159872 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:35.160953 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:35.161995 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:35.163988 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:35.164491 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:35.164664 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:35.171446 (+ 6782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.194:54101: BlockingRecv error: recv got EOF from 127.9.238.194:54101 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 26 16:44:35 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:35 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:35.189035 19326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:35.173517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.194:58527)
0926 16:44:35.173810 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:35.173834 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:35.173990 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:35.175157 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:35.175163 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:35.175790 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:35.176030 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:35.176064 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:35.178520 (+ 2456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:35.178525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:35.179842 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:35.179859 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:35.180973 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:35.180977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:35.181146 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:35.182166 (+ 1020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:35.182197 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:35.188422 (+ 6225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111}
W20260926 16:44:35.189543 19325 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:35.174526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:58527 (local address 127.9.238.254:35741)
0926 16:44:35.174759 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:35.174796 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:35.174816 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:44:35.174887 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:35.174891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:35.174964 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:35.175105 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:35.176353 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:35.178263 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:35.180068 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:35.180609 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:35.180708 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:35.189072 (+ 8364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.194:58527: BlockingRecv error: recv got EOF from 127.9.238.194:58527 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260926 16:44:35.190254 19323 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 16:44:35 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:35 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 16:44:35.584580 19325 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:35.564564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:36037 (local address 127.9.238.254:35741)
0926 16:44:35.564938 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:35.564950 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:35.565492 (+ 542us) server_negotiation.cc:407] Connection header received
0926 16:44:35.565720 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:35.565731 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:35.565909 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:35.566135 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:35.567183 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:35.569007 (+ 1824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:35.570679 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:35.571777 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:35.571993 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:35.584142 (+ 12149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.193:36037: BlockingRecv error: recv got EOF from 127.9.238.193:36037 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
I20260926 16:44:35.585486 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:35.564498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.193:36037)
0926 16:44:35.565324 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:35.565353 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:35.565536 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:35.566208 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:35.566213 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:35.566740 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:35.566974 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:35.566984 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:35.569282 (+ 2298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:35.569287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:35.570454 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:35.570472 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:35.572034 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:35.572039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:35.572218 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:35.577615 (+ 5397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:35.577647 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:35.584975 (+ 7328us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":506,"threads_started":1}
W20260926 16:44:35.774964 19460 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.166s user 0.000s sys 0.015s
W20260926 16:44:35.776436 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:34.587700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:56703 (local address 127.9.238.254:35741)
0926 16:44:34.600678 (+ 12978us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:34.600683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:34.600702 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:44:34.600765 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:34.600769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:34.600849 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:34.601001 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:34.603296 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:34.604689 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:34.606372 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:34.606895 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:34.607028 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:35.776099 (+1169071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.195:56703: BlockingRecv error: recv got EOF from 127.9.238.195:56703 (error 108)
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":12850,"thread_start_us":12653,"threads_started":1}
I20260926 16:44:35.776983 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:34.588581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.195:56703)
0926 16:44:34.589951 (+ 1370us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:34.590043 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:34.593753 (+ 3710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:34.601331 (+ 7578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:34.601346 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:34.602208 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:34.603029 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:34.603070 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:34.604841 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:34.604846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:34.606197 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:34.606215 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:34.607377 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:34.607384 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:34.607749 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:34.608823 (+ 1074us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:34.608857 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:35.775896 (+1167039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":840,"thread_start_us":479,"threads_started":1}
W20260926 16:44:35.777882 19456 heartbeater.cc:644] Failed to heartbeat to 127.9.238.254:35741 (0 consecutive failures): Not authorized: Failed to ping master at 127.9.238.254:35741: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Sep 26 16:44:35 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:35 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:35.788817 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:35.778568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.195:50161)
0926 16:44:35.778780 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:35.778796 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:35.778916 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:35.779335 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:35.779340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:35.779772 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:35.779948 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:35.779957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:35.780971 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:35.780978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:35.781926 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:35.781937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:35.782639 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:35.782645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:35.782807 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:35.783589 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:35.783623 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:35.788530 (+ 4907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":73}
W20260926 16:44:35.789009 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:35.778718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:50161 (local address 127.9.238.254:35741)
0926 16:44:35.779082 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:35.779088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:35.779101 (+ 13us) server_negotiation.cc:407] Connection header received
0926 16:44:35.779146 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:35.779150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:35.779205 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:35.779305 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:35.780084 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:35.780835 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:35.782073 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:35.782463 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:35.782585 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:35.788788 (+ 6203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.195:50161: BlockingRecv error: recv got EOF from 127.9.238.195:50161 (error 108)
Metrics: {"server-negotiator.queue_time_us":230}
Sep 26 16:44:35 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:35 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:35.799853 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:35.789938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.195:34229)
0926 16:44:35.790291 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:35.790307 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:35.790438 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:35.790933 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:35.790937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:35.791372 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:35.791637 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:35.791652 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:35.793085 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:35.793089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:35.794135 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:35.794148 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:35.795021 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:35.795027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:35.795179 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:35.795946 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:35.795963 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:35.799476 (+ 3513us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231}
W20260926 16:44:35.800236 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:35.790095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:34229 (local address 127.9.238.254:35741)
0926 16:44:35.790567 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:35.790572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:35.790594 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:44:35.790657 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:35.790661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:35.790741 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:35.790857 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:35.791830 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:35.792913 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:35.794338 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:35.794812 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:35.794925 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:35.799841 (+ 4916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.195:34229: BlockingRecv error: recv got EOF from 127.9.238.195:34229 (error 108)
Metrics: {"server-negotiator.queue_time_us":303}
W20260926 16:44:35.800482 19456 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 16:44:36 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:36 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:36.203161 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:36.191138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.194:41927)
0926 16:44:36.191961 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:36.191982 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:36.192245 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:36.192902 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:36.192911 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:36.193659 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:36.193933 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:36.193950 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:36.195119 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:36.195123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:36.196359 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:36.196372 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:36.197110 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:36.197116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:36.197313 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:36.198112 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:36.198135 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:36.202868 (+ 4733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":506,"threads_started":1}
W20260926 16:44:36.203634 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:36.191316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:41927 (local address 127.9.238.254:35741)
0926 16:44:36.191737 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:36.191745 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:36.192127 (+ 382us) server_negotiation.cc:407] Connection header received
0926 16:44:36.192515 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:36.192520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:36.192670 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:36.192812 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:36.194128 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:36.194962 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:36.196535 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:36.196925 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:36.197030 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:36.203199 (+ 6169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.194:41927: BlockingRecv error: recv got EOF from 127.9.238.194:41927 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Sep 26 16:44:36 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:36 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:36.597036 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:36.586926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.193:35823)
0926 16:44:36.587488 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:36.587507 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:36.587737 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:36.588105 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:36.588109 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:36.588510 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:36.588733 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:36.588744 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:36.589934 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:36.589940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:36.590993 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:36.591005 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:36.591921 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:36.591925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:36.592074 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:36.592767 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:36.592785 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:36.596645 (+ 3860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":322,"threads_started":1}
W20260926 16:44:36.597283 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:36.586955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:35823 (local address 127.9.238.254:35741)
0926 16:44:36.587333 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:36.587340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:36.587654 (+ 314us) server_negotiation.cc:407] Connection header received
0926 16:44:36.587873 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:36.587877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:36.587955 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:36.588057 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:36.588884 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:36.589714 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:36.591201 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:36.591762 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:36.591868 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:36.596938 (+ 5070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.193:35823: BlockingRecv error: recv got EOF from 127.9.238.193:35823 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 26 16:44:36 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:36 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:36.812676 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:36.801513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.195:40559)
0926 16:44:36.802233 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:36.802253 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:36.802411 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:36.802856 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:36.802859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:36.803296 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:36.803464 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:36.803473 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:36.804847 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:36.804853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:36.805892 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:36.805904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:36.806873 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:36.806878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:36.807038 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:36.808066 (+ 1028us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:36.808089 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:36.812407 (+ 4318us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":415,"threads_started":1}
W20260926 16:44:36.813186 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:36.801649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:40559 (local address 127.9.238.254:35741)
0926 16:44:36.802018 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:36.802027 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:36.802404 (+ 377us) server_negotiation.cc:407] Connection header received
0926 16:44:36.802561 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:36.802567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:36.802684 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:36.802811 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:36.803802 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:36.804642 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:36.806083 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:36.806654 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:36.806790 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:36.812720 (+ 5930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.195:40559: BlockingRecv error: recv got EOF from 127.9.238.195:40559 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
Sep 26 16:44:37 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:37 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:37.216475 19466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:37.204416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.194:46421)
0926 16:44:37.205152 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:37.205173 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:37.205467 (+ 294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:37.205890 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:37.205897 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:37.206435 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:37.206670 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:37.206679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:37.207938 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:37.207945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:37.209029 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:37.209041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:37.209858 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:37.209864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:37.210068 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:37.211089 (+ 1021us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:37.211109 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:37.216018 (+ 4909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":418,"threads_started":1}
W20260926 16:44:37.216650 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:37.204627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:46421 (local address 127.9.238.254:35741)
0926 16:44:37.204973 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:37.204982 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:37.205348 (+ 366us) server_negotiation.cc:407] Connection header received
0926 16:44:37.205630 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:37.205637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:37.205721 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:37.205852 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:37.206834 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:37.207789 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:37.209194 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:37.209669 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:37.209828 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:37.216300 (+ 6472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.194:46421: BlockingRecv error: recv got EOF from 127.9.238.194:46421 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
Sep 26 16:44:37 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:37 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:37.609443 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:37.598658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.193:59183)
0926 16:44:37.599369 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:37.599389 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:37.599596 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:37.600142 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:37.600152 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:37.600745 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:37.601029 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:37.601040 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:37.602470 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:37.602479 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:37.603500 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:37.603512 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:37.604380 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:37.604385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:37.604534 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:37.605207 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:37.605226 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:37.609150 (+ 3924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":406,"threads_started":1}
W20260926 16:44:37.609721 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:37.598993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:59183 (local address 127.9.238.254:35741)
0926 16:44:37.599289 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:37.599297 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:37.599534 (+ 237us) server_negotiation.cc:407] Connection header received
0926 16:44:37.599753 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:37.599757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:37.599836 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:37.600096 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:37.601293 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:37.602300 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:37.603736 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:37.604226 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:37.604349 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:37.609419 (+ 5070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.193:59183: BlockingRecv error: recv got EOF from 127.9.238.193:59183 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 26 16:44:37 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:37 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:37.825073 19468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:37.813897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.195:53581)
0926 16:44:37.814653 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:37.814673 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:37.814864 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:37.815375 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:37.815380 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:37.816079 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:37.816326 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:37.816337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:37.817534 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:37.817539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:37.818548 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:37.818561 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:37.819398 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:37.819403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:37.819585 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:37.820324 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:37.820344 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:37.824802 (+ 4458us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":469,"threads_started":1}
W20260926 16:44:37.825714 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:37.813974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:53581 (local address 127.9.238.254:35741)
0926 16:44:37.814221 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:37.814226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:37.814809 (+ 583us) server_negotiation.cc:407] Connection header received
0926 16:44:37.815055 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:37.815059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:37.815176 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:37.815306 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:37.816492 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:37.817389 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:37.818705 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:37.819198 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:37.819305 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:37.825133 (+ 5828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.195:53581: BlockingRecv error: recv got EOF from 127.9.238.195:53581 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Sep 26 16:44:38 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:38 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:38.228648 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:38.217935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.194:33195)
0926 16:44:38.218628 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:38.218647 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:38.218856 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:38.219229 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:38.219235 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:38.219792 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:38.220016 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:38.220026 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:38.221069 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:38.221075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:38.221990 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:38.222001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:38.222702 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:38.222708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:38.222895 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:38.223663 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:38.223680 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:38.228330 (+ 4650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":414,"threads_started":1}
W20260926 16:44:38.228973 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:38.218044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:33195 (local address 127.9.238.254:35741)
0926 16:44:38.218279 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:38.218288 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:38.218775 (+ 487us) server_negotiation.cc:407] Connection header received
0926 16:44:38.219005 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:38.219011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:38.219081 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:38.219195 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:38.220168 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:38.220924 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:38.222143 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:38.222546 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:38.222681 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:38.228581 (+ 5900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.194:33195: BlockingRecv error: recv got EOF from 127.9.238.194:33195 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 26 16:44:38 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:38 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:38.620841 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:38.610796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.193:35335)
0926 16:44:38.611676 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:38.611704 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:38.611866 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:38.612212 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:38.612219 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:38.612641 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:38.612808 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:38.612817 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:38.613784 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:38.613792 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:38.614784 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:38.614794 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:38.615472 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:38.615478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:38.615689 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:38.616368 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:38.616384 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:38.620460 (+ 4076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":523,"threads_started":1}
W20260926 16:44:38.620937 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:38.611057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:35335 (local address 127.9.238.254:35741)
0926 16:44:38.611326 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:38.611334 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:38.611789 (+ 455us) server_negotiation.cc:407] Connection header received
0926 16:44:38.611988 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:38.611993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:38.612060 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:38.612164 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:38.612948 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:38.613655 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:38.614922 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:38.615325 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:38.615456 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:38.620682 (+ 5226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.193:35335: BlockingRecv error: recv got EOF from 127.9.238.193:35335 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
Sep 26 16:44:38 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:38 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:38.836688 19471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:38.826467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.195:51205)
0926 16:44:38.827170 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:38.827187 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:38.827333 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:38.827828 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:38.827835 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:38.828261 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:38.828460 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:38.828470 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:38.829568 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:38.829575 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:38.830486 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:38.830497 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:38.831221 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:38.831227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:38.831422 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:38.832097 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:38.832117 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:38.836289 (+ 4172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":413,"threads_started":1}
W20260926 16:44:38.836798 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:38.826597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:51205 (local address 127.9.238.254:35741)
0926 16:44:38.826836 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:38.826841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:38.827283 (+ 442us) server_negotiation.cc:407] Connection header received
0926 16:44:38.827469 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:38.827473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:38.827578 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:38.827751 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:38.828626 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:38.829419 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:38.830632 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:38.831032 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:38.831188 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:38.836552 (+ 5364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.195:51205: BlockingRecv error: recv got EOF from 127.9.238.195:51205 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 26 16:44:39 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:39 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:39.241837 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:39.229970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.194:33769)
0926 16:44:39.230655 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:39.230674 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:39.230872 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:39.231257 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:39.231262 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:39.231844 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:39.232041 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:39.232050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:39.233540 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:39.233549 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:39.234676 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:39.234689 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:39.235515 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:39.235521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:39.235760 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:39.236590 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:39.236614 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:39.241563 (+ 4949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":366,"threads_started":1}
W20260926 16:44:39.242353 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:39.230127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:33769 (local address 127.9.238.254:35741)
0926 16:44:39.230425 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:39.230432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:39.230784 (+ 352us) server_negotiation.cc:407] Connection header received
0926 16:44:39.231019 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:39.231025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:39.231104 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:39.231216 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:39.232196 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:39.233320 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:39.234846 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:39.235317 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:39.235498 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:39.241921 (+ 6423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.194:33769: BlockingRecv error: recv got EOF from 127.9.238.194:33769 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Sep 26 16:44:39 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:39 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:39.633570 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:39.622163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.193:39873)
0926 16:44:39.622867 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:39.622888 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:39.623127 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:39.623650 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:39.623656 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:39.624307 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:39.624499 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:39.624510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:39.625896 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:39.625904 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:39.627001 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:39.627015 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:39.627937 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:39.627944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:39.628155 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:39.628907 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:39.628928 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:39.633165 (+ 4237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":389,"threads_started":1}
W20260926 16:44:39.633687 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:39.622291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:39873 (local address 127.9.238.254:35741)
0926 16:44:39.622649 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:39.622657 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:39.623024 (+ 367us) server_negotiation.cc:407] Connection header received
0926 16:44:39.623358 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:39.623363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:39.623440 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:39.623601 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:39.624679 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:39.625733 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:39.627163 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:39.627751 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:39.627903 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:39.633438 (+ 5535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.193:39873: BlockingRecv error: recv got EOF from 127.9.238.193:39873 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Sep 26 16:44:39 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:39 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:39.849263 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:39.838233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.195:36455)
0926 16:44:39.839002 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:39.839023 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:39.839321 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:39.839787 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:39.839792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:39.840274 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:39.840470 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:39.840479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:39.841731 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:39.841740 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:39.842933 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:39.842952 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:39.843758 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:39.843764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:39.843955 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:39.844653 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:39.844672 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:39.848849 (+ 4177us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":424,"threads_started":1}
W20260926 16:44:39.849455 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:39.838382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:36455 (local address 127.9.238.254:35741)
0926 16:44:39.838737 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:39.838746 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:39.839147 (+ 401us) server_negotiation.cc:407] Connection header received
0926 16:44:39.839497 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:39.839501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:39.839619 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:39.839776 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:39.840631 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:39.841595 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:39.843080 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:39.843603 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:39.843740 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:39.849103 (+ 5363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.195:36455: BlockingRecv error: recv got EOF from 127.9.238.195:36455 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Sep 26 16:44:40 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:40 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:40.254805 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:40.243328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.194:53249)
0926 16:44:40.244414 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:40.244435 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:40.244580 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:40.245025 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:40.245033 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:40.245593 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:40.245780 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:40.245791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:40.246919 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:40.246929 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:40.248009 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:40.248021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:40.248863 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:40.248869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:40.249072 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:40.249759 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:40.249778 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:40.254519 (+ 4741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":951,"thread_start_us":809,"threads_started":1}
W20260926 16:44:40.255095 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:40.243671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:53249 (local address 127.9.238.254:35741)
0926 16:44:40.243918 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:40.243925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:40.244552 (+ 627us) server_negotiation.cc:407] Connection header received
0926 16:44:40.244739 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:40.244750 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:40.244827 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:40.244975 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:40.245949 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:40.246767 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:40.248178 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:40.248685 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:40.248835 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:40.254743 (+ 5908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.194:53249: BlockingRecv error: recv got EOF from 127.9.238.194:53249 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 16:44:40 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:40 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:40.652292 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:40.635145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.193:60975)
0926 16:44:40.635904 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:40.635928 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:40.636127 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:40.636663 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:40.636669 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:40.637258 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:40.637463 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:40.637474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:40.639219 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:40.639223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:40.640315 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:40.640328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:40.641215 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:40.641220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:40.641360 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:40.647155 (+ 5795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:40.647179 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:40.652008 (+ 4829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":488,"threads_started":1}
W20260926 16:44:40.652707 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:40.635352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:60975 (local address 127.9.238.254:35741)
0926 16:44:40.635592 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:40.635599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:40.636110 (+ 511us) server_negotiation.cc:407] Connection header received
0926 16:44:40.636310 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:40.636317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:40.636426 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:40.636578 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:40.637738 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:40.639037 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:40.640499 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:40.640992 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:40.641115 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:40.652385 (+ 11270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.193:60975: BlockingRecv error: recv got EOF from 127.9.238.193:60975 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 16:44:40 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:40 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:40.861379 19477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:40.850730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.195:37159)
0926 16:44:40.851462 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:40.851481 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:40.851698 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:40.852119 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:40.852124 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:40.852679 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:40.852840 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:40.852848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:40.853868 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:40.853875 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:40.854814 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:40.854826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:40.855524 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:40.855539 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:40.855749 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:40.856530 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:40.856557 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:40.861102 (+ 4545us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":392,"threads_started":1}
W20260926 16:44:40.861693 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:40.850862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:37159 (local address 127.9.238.254:35741)
0926 16:44:40.851157 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:40.851167 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:40.851598 (+ 431us) server_negotiation.cc:407] Connection header received
0926 16:44:40.851841 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:40.851844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:40.851922 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:40.852078 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:40.852986 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:40.853733 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:40.854955 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:40.855369 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:40.855499 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:40.861391 (+ 5892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.195:37159: BlockingRecv error: recv got EOF from 127.9.238.195:37159 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 26 16:44:41 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:41 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:41.267164 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:41.255890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.194:41129)
0926 16:44:41.256611 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:41.256630 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:41.256832 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:41.257246 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:41.257251 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:41.257767 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:41.257952 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:41.257963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:41.259235 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:41.259241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:41.260451 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:41.260465 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:41.261274 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:41.261280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:41.261491 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:41.262337 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:41.262358 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:41.266771 (+ 4413us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":370,"threads_started":1}
W20260926 16:44:41.267302 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:41.256142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:41129 (local address 127.9.238.254:35741)
0926 16:44:41.256520 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:41.256527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:41.256734 (+ 207us) server_negotiation.cc:407] Connection header received
0926 16:44:41.257015 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:41.257020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:41.257091 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:41.257203 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:41.258105 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:41.259038 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:41.260617 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:41.261088 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:41.261234 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:41.267018 (+ 5784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.194:41129: BlockingRecv error: recv got EOF from 127.9.238.194:41129 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Sep 26 16:44:41 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:41 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:41.665035 19479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:41.653637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.193:55237)
0926 16:44:41.654324 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:41.654343 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:41.654482 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:41.654842 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:41.654846 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:41.655307 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:41.655507 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:41.655516 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:41.656648 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:41.656653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:41.658086 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:41.658101 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:41.659068 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:41.659072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:41.659203 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:41.660129 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:41.660150 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:41.664655 (+ 4505us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":332,"threads_started":1}
W20260926 16:44:41.665083 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:41.653677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:55237 (local address 127.9.238.254:35741)
0926 16:44:41.653880 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:41.653884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:41.654463 (+ 579us) server_negotiation.cc:407] Connection header received
0926 16:44:41.654597 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:41.654601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:41.654673 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:41.654768 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:41.655746 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:41.656460 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:41.658301 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:41.658890 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:41.659031 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:41.664842 (+ 5811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.193:55237: BlockingRecv error: recv got EOF from 127.9.238.193:55237 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 26 16:44:41 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:41 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:41.873060 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:41.862720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.195:56321)
0926 16:44:41.863430 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:41.863449 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:41.863613 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:41.863998 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:41.864001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:41.864459 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:41.864633 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:41.864642 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:41.865826 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:41.865830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:41.866775 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:41.866787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:41.867708 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:41.867713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:41.867844 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:41.868589 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:41.868609 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:41.872645 (+ 4036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":388,"threads_started":1}
W20260926 16:44:41.873452 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:41.862798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:56321 (local address 127.9.238.254:35741)
0926 16:44:41.863083 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:41.863093 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:41.863540 (+ 447us) server_negotiation.cc:407] Connection header received
0926 16:44:41.863725 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:41.863729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:41.863843 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:41.863949 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:41.864809 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:41.865657 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:41.866936 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:41.867459 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:41.867683 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:41.872932 (+ 5249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.195:56321: BlockingRecv error: recv got EOF from 127.9.238.195:56321 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 26 16:44:42 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:42 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:42.279515 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:42.268648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.194:33381)
0926 16:44:42.269666 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:42.269692 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:42.269859 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:42.270348 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:42.270353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:42.270995 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:42.271205 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:42.271215 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:42.272429 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:42.272437 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:42.273511 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:42.273528 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:42.274327 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:42.274334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:42.274542 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:42.275267 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:42.275287 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:42.279115 (+ 3828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":855,"thread_start_us":533,"threads_started":1}
W20260926 16:44:42.279573 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:42.268865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:33381 (local address 127.9.238.254:35741)
0926 16:44:42.269116 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:42.269126 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:42.269781 (+ 655us) server_negotiation.cc:407] Connection header received
0926 16:44:42.270072 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:42.270077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:42.270157 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:42.270301 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:42.271370 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:42.272272 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:42.273672 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:42.274151 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:42.274305 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:42.279325 (+ 5020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.194:33381: BlockingRecv error: recv got EOF from 127.9.238.194:33381 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 26 16:44:42 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:42 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:42.677256 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:42.666443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.193:36117)
0926 16:44:42.666996 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:42.667013 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:42.667142 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:42.667639 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:42.667642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:42.668015 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:42.668237 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:42.668248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:42.669525 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:42.669530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:42.670495 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:42.670506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:42.671456 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:42.671461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:42.671700 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:42.672682 (+ 982us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:42.672706 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:42.676924 (+ 4218us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":290,"threads_started":1}
W20260926 16:44:42.677416 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:42.666662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:36117 (local address 127.9.238.254:35741)
0926 16:44:42.666898 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:42.666903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:42.667131 (+ 228us) server_negotiation.cc:407] Connection header received
0926 16:44:42.667334 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:42.667338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:42.667420 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:42.667541 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:42.668468 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:42.669357 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:42.670644 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:42.671210 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:42.671417 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:42.677173 (+ 5756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.193:36117: BlockingRecv error: recv got EOF from 127.9.238.193:36117 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 26 16:44:42 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:42 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:42.886801 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:42.874070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.195:48431)
0926 16:44:42.874910 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:42.874932 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:42.875155 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:42.875612 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:42.875618 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:42.876183 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:42.876390 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:42.876401 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:42.877755 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:42.877761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:42.879062 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:42.879079 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:42.880066 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:42.880073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:42.880274 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:42.881169 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:42.881191 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:42.886376 (+ 5185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":441,"threads_started":1}
W20260926 16:44:42.886849 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:42.874391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:48431 (local address 127.9.238.254:35741)
0926 16:44:42.874732 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:42.874740 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:42.875062 (+ 322us) server_negotiation.cc:407] Connection header received
0926 16:44:42.875309 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:42.875321 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:42.875410 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:42.875537 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:42.876563 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:42.877538 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:42.879259 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:42.879860 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:42.880017 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:42.886566 (+ 6549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.195:48431: BlockingRecv error: recv got EOF from 127.9.238.195:48431 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
Sep 26 16:44:43 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:43 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:43.291746 19484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:43.281229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.194:53673)
0926 16:44:43.282109 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:43.282128 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:43.282320 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:43.282818 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:43.282823 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:43.283311 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:43.283494 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:43.283503 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.284997 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.285000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:43.285942 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:43.285952 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.286803 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.286807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:43.286920 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:43.287637 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:43.287661 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:43.291385 (+ 3724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":356,"threads_started":1}
W20260926 16:44:43.292117 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:43.281379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:53673 (local address 127.9.238.254:35741)
0926 16:44:43.281689 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:43.281698 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:43.282240 (+ 542us) server_negotiation.cc:407] Connection header received
0926 16:44:43.282497 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:43.282503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:43.282608 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:43.282760 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:43.283706 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.284858 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.286104 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.286628 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.286744 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:43.291801 (+ 5057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.194:53673: BlockingRecv error: recv got EOF from 127.9.238.194:53673 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 26 16:44:43 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:43 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:43.690425 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:43.678657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.193:48989)
0926 16:44:43.679331 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:43.679378 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:43.679586 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:43.680095 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:43.680101 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:43.680706 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:43.680863 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:43.680872 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.681999 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.682003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:43.683044 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:43.683095 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.684192 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.684196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:43.684341 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:43.685093 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:43.685114 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:43.690059 (+ 4945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":530,"threads_started":1}
W20260926 16:44:43.690771 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:43.678900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:48989 (local address 127.9.238.254:35741)
0926 16:44:43.679219 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:43.679226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:43.679487 (+ 261us) server_negotiation.cc:407] Connection header received
0926 16:44:43.679768 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:43.679773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:43.679878 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:43.680019 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:43.681011 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.681840 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.683243 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.684008 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.684157 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:43.690419 (+ 6262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.193:48989: BlockingRecv error: recv got EOF from 127.9.238.193:48989 (error 108)
Metrics: {"server-negotiator.queue_time_us":151}
Sep 26 16:44:43 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:43 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:43.898538 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:43.887991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.195:57643)
0926 16:44:43.888583 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:43.888600 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:43.888731 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:43.889108 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:43.889112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:43.889519 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:43.889822 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:43.889832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.890922 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.890926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:43.892081 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:43.892094 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.892872 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.892876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:43.892999 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:43.893750 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:43.893768 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:43.898155 (+ 4387us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":322,"threads_started":1}
W20260926 16:44:43.898773 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:43.888097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:57643 (local address 127.9.238.254:35741)
0926 16:44:43.888295 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:43.888300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:43.888680 (+ 380us) server_negotiation.cc:407] Connection header received
0926 16:44:43.888844 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:43.888847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:43.888960 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:43.889060 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:43.890002 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.890788 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.892237 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:43.892704 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:43.892810 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:43.898448 (+ 5638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.195:57643: BlockingRecv error: recv got EOF from 127.9.238.195:57643 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 16:44:44 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:44 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:44.303632 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:44.292922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.194:49151)
0926 16:44:44.293512 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:44.293530 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:44.293691 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:44.294106 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:44.294114 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:44.294545 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:44.294739 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:44.294751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:44.296523 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:44.296527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:44.297510 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:44.297522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:44.298304 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:44.298309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:44.298447 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:44.299181 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:44.299200 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:44.303273 (+ 4073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":320,"threads_started":1}
W20260926 16:44:44.304119 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:44.293106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:49151 (local address 127.9.238.254:35741)
0926 16:44:44.293315 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:44.293320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:44.293650 (+ 330us) server_negotiation.cc:407] Connection header received
0926 16:44:44.293830 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:44.293834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:44.293911 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:44.294031 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:44.294964 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:44.296360 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:44.297667 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:44.298119 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:44.298224 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:44.303795 (+ 5571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.194:49151: BlockingRecv error: recv got EOF from 127.9.238.194:49151 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 16:44:44 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:44 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:44.702190 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:44.691622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.193:46267)
0926 16:44:44.692249 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:44.692267 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:44.692417 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:44.692786 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:44.692789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:44.693193 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:44.693378 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:44.693388 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:44.694520 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:44.694524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:44.695845 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:44.695858 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:44.696907 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:44.696923 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:44.697118 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:44.697922 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:44.697941 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:44.701908 (+ 3967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":356,"threads_started":1}
W20260926 16:44:44.702438 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:44.691755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:46267 (local address 127.9.238.254:35741)
0926 16:44:44.691940 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:44.691945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:44.692365 (+ 420us) server_negotiation.cc:407] Connection header received
0926 16:44:44.692556 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:44.692560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:44.692631 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:44.692741 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:44.693578 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:44.694339 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:44.696076 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:44.696709 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:44.696874 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:44.702135 (+ 5261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.193:46267: BlockingRecv error: recv got EOF from 127.9.238.193:46267 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 16:44:44 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:44 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:44.910338 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:44.899731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.195:49173)
0926 16:44:44.900559 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:44.900609 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:44.900790 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:44.901384 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:44.901390 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:44.901926 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:44.902188 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:44.902199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:44.903187 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:44.903195 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:44.904283 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:44.904295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:44.905070 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:44.905076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:44.905246 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:44.905942 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:44.905960 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:44.909962 (+ 4002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":579,"threads_started":1}
W20260926 16:44:44.910562 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:44.899992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:49173 (local address 127.9.238.254:35741)
0926 16:44:44.900273 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:44.900277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:44.900775 (+ 498us) server_negotiation.cc:407] Connection header received
0926 16:44:44.900936 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:44.900943 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:44.901148 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:44.901357 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:44.902342 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:44.903046 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:44.904442 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:44.904870 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:44.905023 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:44.910262 (+ 5239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.195:49173: BlockingRecv error: recv got EOF from 127.9.238.195:49173 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 16:44:45 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:45 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:45.314756 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:45.304693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.194:42199)
0926 16:44:45.305371 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:45.305389 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:45.305527 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:45.305879 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:45.305883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:45.306306 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:45.306487 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:45.306496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:45.307509 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:45.307513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:45.308622 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:45.308634 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:45.309388 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:45.309393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:45.309528 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:45.310320 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:45.310339 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:45.314350 (+ 4011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":411,"threads_started":1}
W20260926 16:44:45.314965 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:45.304858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:42199 (local address 127.9.238.254:35741)
0926 16:44:45.305072 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:45.305079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:45.305482 (+ 403us) server_negotiation.cc:407] Connection header received
0926 16:44:45.305660 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:45.305663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:45.305729 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:45.305829 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:45.306655 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:45.307356 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:45.308758 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:45.309212 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:45.309317 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:45.314638 (+ 5321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.194:42199: BlockingRecv error: recv got EOF from 127.9.238.194:42199 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 16:44:45 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:45 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:45.723721 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:45.703640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.193:38085)
0926 16:44:45.704341 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:45.704368 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:45.704577 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:45.705157 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:45.705162 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:45.705763 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:45.706009 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:45.706019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:45.707089 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:45.707097 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:45.708338 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:45.708350 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:45.709067 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:45.709073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:45.709262 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:45.718950 (+ 9688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:45.718977 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:45.723298 (+ 4321us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":354,"threads_started":1}
W20260926 16:44:45.723897 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:45.703899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:38085 (local address 127.9.238.254:35741)
0926 16:44:45.704519 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:45.704526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:45.704548 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:44:45.704898 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:45.704902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:45.704972 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:45.705094 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:45.706173 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:45.706952 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:45.708489 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:45.708903 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:45.709044 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:45.723601 (+ 14557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.193:38085: BlockingRecv error: recv got EOF from 127.9.238.193:38085 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
Sep 26 16:44:45 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:45 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:45.921401 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:45.911477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.195:34619)
0926 16:44:45.912120 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:45.912139 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:45.912313 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:45.912705 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:45.912709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:45.913163 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:45.913324 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:45.913332 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:45.914364 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:45.914371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:45.915332 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:45.915343 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:45.916154 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:45.916160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:45.916317 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:45.916979 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:45.916995 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:45.921003 (+ 4008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":362,"threads_started":1}
W20260926 16:44:45.921531 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:45.911649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:34619 (local address 127.9.238.254:35741)
0926 16:44:45.911912 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:45.911922 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:45.912233 (+ 311us) server_negotiation.cc:407] Connection header received
0926 16:44:45.912450 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:45.912455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:45.912543 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:45.912683 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:45.913463 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:45.914234 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:45.915481 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:45.915992 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:45.916125 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:45.921216 (+ 5091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.195:34619: BlockingRecv error: recv got EOF from 127.9.238.195:34619 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 26 16:44:46 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:46 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:46.329674 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.316238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.194:35789)
0926 16:44:46.317049 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.317073 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:46.317298 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:46.317898 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:46.317904 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:46.318607 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:46.319000 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:46.319014 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.320647 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.320653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:46.322453 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:46.322468 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.323485 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.323492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:46.323804 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.324620 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:46.324640 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:46.329288 (+ 4648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":2670,"threads_started":1}
W20260926 16:44:46.329877 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:46.317343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:35789 (local address 127.9.238.254:35741)
0926 16:44:46.317571 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:46.317576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:46.317594 (+ 18us) server_negotiation.cc:407] Connection header received
0926 16:44:46.317655 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:46.317659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:46.317727 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:46.317857 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:46.319202 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.320435 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.322639 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.323242 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.323438 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.329632 (+ 6194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.194:35789: BlockingRecv error: recv got EOF from 127.9.238.194:35789 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 26 16:44:46 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:46 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:46.736825 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.725228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.193:49905)
0926 16:44:46.725917 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.725936 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:46.726084 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:46.726563 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:46.726567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:46.727041 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:46.727243 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:46.727253 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.728487 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.728492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:46.729658 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:46.729679 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.730677 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.730685 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:46.730962 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.731876 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:46.731905 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:46.736484 (+ 4579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":405,"threads_started":1}
W20260926 16:44:46.737030 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:46.725414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:49905 (local address 127.9.238.254:35741)
0926 16:44:46.725687 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:46.725693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:46.726237 (+ 544us) server_negotiation.cc:407] Connection header received
0926 16:44:46.726313 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:46.726317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:46.726390 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:46.726506 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:46.727435 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.728305 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.729878 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.730436 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.730612 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.736732 (+ 6120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.193:49905: BlockingRecv error: recv got EOF from 127.9.238.193:49905 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
Sep 26 16:44:46 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:46 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:46.935756 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:46.922926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.195:45869)
0926 16:44:46.923804 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:46.923842 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:46.924053 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:46.924525 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:46.924531 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:46.925101 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:46.925339 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:46.925349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.926529 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.926538 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:46.927704 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:46.927716 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.928577 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.928583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:46.928807 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.929555 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:46.929580 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:46.935289 (+ 5709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":737,"thread_start_us":472,"threads_started":1}
W20260926 16:44:46.936014 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:46.923201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:45869 (local address 127.9.238.254:35741)
0926 16:44:46.923611 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:46.923621 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:46.923948 (+ 327us) server_negotiation.cc:407] Connection header received
0926 16:44:46.924261 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:46.924267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:46.924354 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:46.924487 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:46.925511 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.926373 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.927874 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:46.928386 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:46.928535 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:46.935717 (+ 7182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.195:45869: BlockingRecv error: recv got EOF from 127.9.238.195:45869 (error 108)
Metrics: {"server-negotiator.queue_time_us":145}
Sep 26 16:44:47 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:47 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:47.341351 19496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:47.331126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.194:60499)
0926 16:44:47.331743 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:47.331761 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:47.331899 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:47.332321 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:47.332325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:47.332798 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:47.332987 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:47.332997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:47.334119 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:47.334124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:47.335105 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:47.335118 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:47.336003 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:47.336007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:47.336164 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:47.336904 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:47.336924 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:47.340959 (+ 4035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":317,"threads_started":1}
W20260926 16:44:47.341588 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:47.331266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:60499 (local address 127.9.238.254:35741)
0926 16:44:47.331514 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:47.331519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:47.331848 (+ 329us) server_negotiation.cc:407] Connection header received
0926 16:44:47.332058 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:47.332062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:47.332136 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:47.332240 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:47.333132 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:47.333970 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:47.335293 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:47.335843 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:47.335971 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:47.341287 (+ 5316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.194:60499: BlockingRecv error: recv got EOF from 127.9.238.194:60499 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
Sep 26 16:44:47 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:47 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:47.749341 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:47.738253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.193:56451)
0926 16:44:47.738821 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:47.738840 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:47.739006 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:47.739413 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:47.739417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:47.739858 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:47.740025 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:47.740033 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:47.741226 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:47.741230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:47.742184 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:47.742195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:47.742984 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:47.742988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:47.743155 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:47.744007 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:47.744035 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:47.748957 (+ 4922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":318,"threads_started":1}
W20260926 16:44:47.749537 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:47.738360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:56451 (local address 127.9.238.254:35741)
0926 16:44:47.738646 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:47.738654 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:47.738980 (+ 326us) server_negotiation.cc:407] Connection header received
0926 16:44:47.739150 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:47.739154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:47.739246 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:47.739360 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:47.740210 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:47.741050 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:47.742380 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:47.742835 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:47.742961 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:47.749192 (+ 6231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.193:56451: BlockingRecv error: recv got EOF from 127.9.238.193:56451 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 26 16:44:47 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:47 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:47.948041 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:47.937400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.195:36969)
0926 16:44:47.938183 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:47.938208 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:47.938381 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:47.938857 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:47.938864 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:47.939526 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:47.939766 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:47.939775 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:47.941134 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:47.941139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:47.942156 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:47.942168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:47.942889 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:47.942894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:47.943099 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:47.943860 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:47.943880 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:47.947650 (+ 3770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":359,"threads_started":1}
W20260926 16:44:47.948419 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:47.937485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:36969 (local address 127.9.238.254:35741)
0926 16:44:47.937783 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:47.937794 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:47.938320 (+ 526us) server_negotiation.cc:407] Connection header received
0926 16:44:47.938562 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:47.938568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:47.938675 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:47.938779 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:47.939942 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:47.940928 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:47.942300 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:47.942709 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:47.942809 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:47.948051 (+ 5242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.195:36969: BlockingRecv error: recv got EOF from 127.9.238.195:36969 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 26 16:44:48 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:48 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:48.354732 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:48.342750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.194:33521)
0926 16:44:48.343514 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:48.343535 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:48.343756 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:48.344146 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:48.344151 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:48.344633 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:48.344815 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:48.344824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:48.345920 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:48.345927 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:48.346892 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:48.346904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:48.348105 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:48.348116 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:48.348415 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:48.349393 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:48.349424 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:48.354467 (+ 5043us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":482,"threads_started":1}
W20260926 16:44:48.355183 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:48.342949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:33521 (local address 127.9.238.254:35741)
0926 16:44:48.343180 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:48.343190 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:48.343661 (+ 471us) server_negotiation.cc:407] Connection header received
0926 16:44:48.343893 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:48.343899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:48.343991 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:48.344102 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:48.344962 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:48.345773 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:48.347039 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:48.347847 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:48.348007 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:48.354870 (+ 6863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.194:33521: BlockingRecv error: recv got EOF from 127.9.238.194:33521 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 16:44:48 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:48 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.193@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:48.762025 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:48.751164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.193:44281)
0926 16:44:48.751851 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:48.751870 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:48.752071 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:48.752521 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:48.752526 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:48.753095 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:48.753292 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:48.753302 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:48.754508 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:48.754515 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:48.755508 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:48.755520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:48.756381 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:48.756387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:48.756599 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:48.757356 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:48.757375 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:48.761620 (+ 4245us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":319,"threads_started":1}
W20260926 16:44:48.762144 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:48.751423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:44281 (local address 127.9.238.254:35741)
0926 16:44:48.751719 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:48.751727 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:48.751984 (+ 257us) server_negotiation.cc:407] Connection header received
0926 16:44:48.752268 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:48.752274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:48.752358 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:48.752490 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:48.753448 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:48.754349 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:48.755723 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:48.756213 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:48.756350 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:48.761855 (+ 5505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.193:44281: BlockingRecv error: recv got EOF from 127.9.238.193:44281 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 26 16:44:48 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:48 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.195@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:48.960268 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:48.949450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.195:55889)
0926 16:44:48.950029 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:48.950047 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:48.950188 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:48.950574 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:48.950577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:48.951050 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:48.951329 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:48.951342 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:48.952535 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:48.952539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:48.953605 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:48.953617 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:48.954648 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:48.954676 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:48.954856 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:48.955736 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:48.955756 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:48.960008 (+ 4252us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":334,"threads_started":1}
W20260926 16:44:48.960559 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:48.949560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:55889 (local address 127.9.238.254:35741)
0926 16:44:48.949858 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:48.949868 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:48.950154 (+ 286us) server_negotiation.cc:407] Connection header received
0926 16:44:48.950322 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:48.950329 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:48.950403 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:48.950520 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:48.951469 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:48.952413 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:48.953760 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:48.954487 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:48.954629 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:48.960266 (+ 5637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.195:55889: BlockingRecv error: recv got EOF from 127.9.238.195:55889 (error 108)
Metrics: {"server-negotiator.queue_time_us":155}
Sep 26 16:44:49 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 16:44:49 dist-test-slave-wt2j krb5kdc[18454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.9.238.194@KRBTEST.COM for oryx/127.9.238.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 16:44:49.370532 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:49.356423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35741 (local address 127.9.238.194:54799)
0926 16:44:49.360107 (+ 3684us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:49.360127 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:49.360278 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:49.360798 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:49.360802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:49.361243 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:49.361463 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:49.361472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:49.362675 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:49.362680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:49.363984 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:49.363997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:49.364838 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:49.364842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:49.364981 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:49.365714 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:49.365756 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:49.370132 (+ 4376us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.9.238.254:35741: Server oryx/127.9.238.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3477,"thread_start_us":3299,"threads_started":1}
W20260926 16:44:49.370628 19459 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:49.356519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:54799 (local address 127.9.238.254:35741)
0926 16:44:49.356807 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:49.356814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:49.360236 (+ 3422us) server_negotiation.cc:407] Connection header received
0926 16:44:49.360398 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:49.360402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:49.360535 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:49.360712 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:49.361635 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:49.362490 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:49.364130 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:49.364657 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:49.364767 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:49.370363 (+ 5596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.9.238.194:54799: BlockingRecv error: recv got EOF from 127.9.238.194:54799 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
I20260926 16:44:49.586725 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 19043
I20260926 16:44:49.601559 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:49.588167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36492 (local address 127.9.238.193:34485)
0926 16:44:49.588964 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:49.588974 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:49.589010 (+ 36us) server_negotiation.cc:407] Connection header received
0926 16:44:49.589102 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:49.589110 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:49.589427 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:49.589639 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:49.591230 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:49.592528 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:49.593951 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:49.594604 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:49.594731 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:49.597658 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:49.597670 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:49.597690 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:49.597757 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:49.600887 (+ 3130us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.238.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 16:44:49.601260 (+ 373us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36492: Request ticket server kudu/127.9.238.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":415,"threads_started":1}
W20260926 16:44:49.601686 19504 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36492: Request ticket server kudu/127.9.238.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 16:44:49.601775 10171 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 19043: Not authorized: Client connection negotiation failed: client connection to 127.9.238.193:34485: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.238.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 16:44:49.602255 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 19043
I20260926 16:44:49.622162 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 19194
I20260926 16:44:49.636992 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:49.623772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45896 (local address 127.9.238.194:35017)
0926 16:44:49.624668 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:49.624680 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:49.624726 (+ 46us) server_negotiation.cc:407] Connection header received
0926 16:44:49.624844 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:49.624857 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:49.625277 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:49.625572 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:49.627104 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:49.628450 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:49.629733 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:49.630377 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:49.630490 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:49.633146 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:49.633156 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:49.633173 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:49.633218 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:49.636283 (+ 3065us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.238.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 16:44:49.636640 (+ 357us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45896: Request ticket server kudu/127.9.238.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":417,"threads_started":1}
W20260926 16:44:49.637117 19505 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45896: Request ticket server kudu/127.9.238.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 16:44:49.637157 10171 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 19194: Not authorized: Client connection negotiation failed: client connection to 127.9.238.194:35017: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.238.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 16:44:49.637352 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 19194
I20260926 16:44:49.654891 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 19327
I20260926 16:44:49.673518 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:49.656483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50574 (local address 127.9.238.195:42115)
0926 16:44:49.660241 (+ 3758us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:49.660250 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:49.660297 (+ 47us) server_negotiation.cc:407] Connection header received
0926 16:44:49.660389 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:49.660397 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:49.660721 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:49.660965 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:49.662583 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:49.664111 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:49.665448 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:49.666103 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:49.666220 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:49.669079 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:49.669089 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:49.669108 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:49.669182 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:49.672742 (+ 3560us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.238.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0926 16:44:49.673133 (+ 391us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50574: Request ticket server kudu/127.9.238.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":3492,"thread_start_us":3310,"threads_started":1}
W20260926 16:44:49.673653 19506 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50574: Request ticket server kudu/127.9.238.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260926 16:44:49.673677 10171 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 19327: Not authorized: Client connection negotiation failed: client connection to 127.9.238.195:42115: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.9.238.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260926 16:44:49.673924 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 19327
I20260926 16:44:49.692625 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 18974
I20260926 16:44:49.694319 19036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:49.814160 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 18974
2026-09-26T16:44:49Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (22195 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T16:44:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:44:49Z Disabled control of system clock
I20260926 16:44:49.908947 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:38215
--webserver_interface=127.9.238.254
--webserver_port=0
--builtin_ntp_servers=127.9.238.212:38275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:38215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 16:44:50.313270 19511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:50.313594 19511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:50.313681 19511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:50.321367 19511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:44:50.321498 19511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:50.321537 19511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:50.321614 19511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:44:50.321696 19511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:44:50.332561 19511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:38275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:38215
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:38215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:50.334103 19511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:50.336262 19511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:50.351821 19516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:50.352016 19517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:50.352340 19519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:50.352401 19511 server_base.cc:1034] running on GCE node
I20260926 16:44:50.353510 19511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:50.355090 19511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:50.356611 19511 hybrid_clock.cc:648] HybridClock initialized: now 1790441090356497 us; error 69 us; skew 500 ppm
I20260926 16:44:50.357149 19511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:44:50.360808 19511 webserver.cc:460] Webserver started at http://127.9.238.254:34473/ using document root <none> and password file <none>
I20260926 16:44:50.361860 19511 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:50.361977 19511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:50.362500 19511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:50.365674 19511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "ea6772e85f2743f8a4d4a87570346ead"
format_stamp: "Formatted at 2026-09-26 16:44:50 on dist-test-slave-wt2j"
server_key: "cfd8e1af68601b8f5123c767499518a6"
server_key_iv: "49761d77bee2bac17bdd13d91ad7ab0e"
server_key_version: "encryptionkey@0"
I20260926 16:44:50.366525 19511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "ea6772e85f2743f8a4d4a87570346ead"
format_stamp: "Formatted at 2026-09-26 16:44:50 on dist-test-slave-wt2j"
server_key: "cfd8e1af68601b8f5123c767499518a6"
server_key_iv: "49761d77bee2bac17bdd13d91ad7ab0e"
server_key_version: "encryptionkey@0"
I20260926 16:44:50.372745 19511 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 16:44:50.376972 19525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:50.379011 19511 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 16:44:50.379261 19511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "ea6772e85f2743f8a4d4a87570346ead"
format_stamp: "Formatted at 2026-09-26 16:44:50 on dist-test-slave-wt2j"
server_key: "cfd8e1af68601b8f5123c767499518a6"
server_key_iv: "49761d77bee2bac17bdd13d91ad7ab0e"
server_key_version: "encryptionkey@0"
I20260926 16:44:50.379550 19511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:50.391171 19511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:50.392377 19511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:50.392771 19511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:50.413283 19511 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:38215
I20260926 16:44:50.413333 19576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:38215 every 8 connection(s)
I20260926 16:44:50.414847 19511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:44:50.419481 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 19511
I20260926 16:44:50.419754 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:44:50.420626 19577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:50.420704 10171 external_mini_cluster.cc:1428] Setting key e5f2cb85424a31a57b09ed4d63bf328c
I20260926 16:44:50.435876 19580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:50.423356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39792 (local address 127.9.238.254:38215)
0926 16:44:50.424792 (+ 1436us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:50.424806 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:50.424858 (+ 52us) server_negotiation.cc:407] Connection header received
0926 16:44:50.426573 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:50.426597 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:50.427156 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:50.427638 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:50.428286 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:50.429683 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:50.431003 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:50.431801 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:50.432011 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:50.433277 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:50.433284 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:50.433298 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:44:50.433942 (+ 644us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:50.434041 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:44:50.434062 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:50.434224 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:50.434420 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 16:44:50.434765 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":813,"thread_start_us":452,"threads_started":1}
I20260926 16:44:50.437415 19577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea6772e85f2743f8a4d4a87570346ead: Bootstrap starting.
I20260926 16:44:50.442059 19577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea6772e85f2743f8a4d4a87570346ead: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:50.443871 19577 log.cc:824] T 00000000000000000000000000000000 P ea6772e85f2743f8a4d4a87570346ead: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:50.448124 19577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea6772e85f2743f8a4d4a87570346ead: No bootstrap required, opened a new log
I20260926 16:44:50.457824 19577 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea6772e85f2743f8a4d4a87570346ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea6772e85f2743f8a4d4a87570346ead" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 38215 } }
I20260926 16:44:50.458380 19577 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea6772e85f2743f8a4d4a87570346ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:50.458501 19577 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea6772e85f2743f8a4d4a87570346ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea6772e85f2743f8a4d4a87570346ead, State: Initialized, Role: FOLLOWER
I20260926 16:44:50.459334 19577 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea6772e85f2743f8a4d4a87570346ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea6772e85f2743f8a4d4a87570346ead" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 38215 } }
I20260926 16:44:50.459781 19577 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea6772e85f2743f8a4d4a87570346ead [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:44:50.460045 19577 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea6772e85f2743f8a4d4a87570346ead [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:44:50.460281 19577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea6772e85f2743f8a4d4a87570346ead [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:50.462822 19577 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea6772e85f2743f8a4d4a87570346ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea6772e85f2743f8a4d4a87570346ead" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 38215 } }
I20260926 16:44:50.463536 19577 leader_election.cc:302] T 00000000000000000000000000000000 P ea6772e85f2743f8a4d4a87570346ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea6772e85f2743f8a4d4a87570346ead; no voters:
I20260926 16:44:50.464293 19577 leader_election.cc:288] T 00000000000000000000000000000000 P ea6772e85f2743f8a4d4a87570346ead [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:44:50.464530 19582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea6772e85f2743f8a4d4a87570346ead [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:44:50.466128 19582 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea6772e85f2743f8a4d4a87570346ead [term 1 LEADER]: Becoming Leader. State: Replica: ea6772e85f2743f8a4d4a87570346ead, State: Running, Role: LEADER
I20260926 16:44:50.466799 19582 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea6772e85f2743f8a4d4a87570346ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea6772e85f2743f8a4d4a87570346ead" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 38215 } }
I20260926 16:44:50.467155 19577 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea6772e85f2743f8a4d4a87570346ead [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:44:50.469638 19584 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea6772e85f2743f8a4d4a87570346ead [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea6772e85f2743f8a4d4a87570346ead. Latest consensus state: current_term: 1 leader_uuid: "ea6772e85f2743f8a4d4a87570346ead" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea6772e85f2743f8a4d4a87570346ead" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 38215 } } }
I20260926 16:44:50.470005 19584 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea6772e85f2743f8a4d4a87570346ead [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:50.470695 19583 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea6772e85f2743f8a4d4a87570346ead [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea6772e85f2743f8a4d4a87570346ead" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea6772e85f2743f8a4d4a87570346ead" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 38215 } } }
I20260926 16:44:50.471231 19583 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea6772e85f2743f8a4d4a87570346ead [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:50.471580 19588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:44:50.477648 19588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:44:50.488416 19588 catalog_manager.cc:1347] Generated new cluster ID: 88dcace292834053b3bb5ccd4f23dcdb
I20260926 16:44:50.488615 19588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:44:50.501927 19588 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:44:50.503412 19588 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:44:50.520484 19588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea6772e85f2743f8a4d4a87570346ead: Generated new TSK 0
I20260926 16:44:50.521937 19588 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:44:50.538955 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:0
--local_ip_for_outbound_sockets=127.9.238.193
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:38215
--builtin_ntp_servers=127.9.238.212:38275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 16:44:50.891969 19601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:50.892259 19601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:50.892395 19601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:50.899778 19601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:50.899912 19601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:50.900075 19601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:44:50.909494 19601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:38275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:38215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:50.910856 19601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:50.912765 19601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:50.927799 19606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:50.927736 19609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:50.927766 19607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:50.928808 19601 server_base.cc:1034] running on GCE node
I20260926 16:44:50.929670 19601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:50.931131 19601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:50.932476 19601 hybrid_clock.cc:648] HybridClock initialized: now 1790441090932398 us; error 55 us; skew 500 ppm
I20260926 16:44:50.932978 19601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:44:50.942517 19601 webserver.cc:460] Webserver started at http://127.9.238.193:35757/ using document root <none> and password file <none>
I20260926 16:44:50.943691 19601 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:50.943867 19601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:50.944294 19601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:50.947921 19601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/data/instance:
uuid: "6623e3b6dc9a485ebe6287d3a95e7142"
format_stamp: "Formatted at 2026-09-26 16:44:50 on dist-test-slave-wt2j"
server_key: "5f06e29bd01c90f4ac46bbf49adfc585"
server_key_iv: "741a94293e03297677aea5ecb359e4a0"
server_key_version: "encryptionkey@0"
I20260926 16:44:50.948848 19601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance:
uuid: "6623e3b6dc9a485ebe6287d3a95e7142"
format_stamp: "Formatted at 2026-09-26 16:44:50 on dist-test-slave-wt2j"
server_key: "5f06e29bd01c90f4ac46bbf49adfc585"
server_key_iv: "741a94293e03297677aea5ecb359e4a0"
server_key_version: "encryptionkey@0"
I20260926 16:44:50.955246 19601 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 16:44:50.959769 19615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:50.961745 19601 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:44:50.961990 19601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "6623e3b6dc9a485ebe6287d3a95e7142"
format_stamp: "Formatted at 2026-09-26 16:44:50 on dist-test-slave-wt2j"
server_key: "5f06e29bd01c90f4ac46bbf49adfc585"
server_key_iv: "741a94293e03297677aea5ecb359e4a0"
server_key_version: "encryptionkey@0"
I20260926 16:44:50.962270 19601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:50.977875 19601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:50.978974 19601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:50.979310 19601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:50.980572 19601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:50.982350 19601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:50.982484 19601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:50.982625 19601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:50.982717 19601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:51.024258 19601 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:42993
I20260926 16:44:51.024284 19727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:42993 every 8 connection(s)
I20260926 16:44:51.025925 19601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:44:51.034467 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 19601
I20260926 16:44:51.034729 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance
I20260926 16:44:51.035687 10171 external_mini_cluster.cc:1428] Setting key 752cc8b1fa36bade866c91deb0f5efaf
I20260926 16:44:51.038908 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:0
--local_ip_for_outbound_sockets=127.9.238.194
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:38215
--builtin_ntp_servers=127.9.238.212:38275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 16:44:51.041715 19730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:51.028950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:36951 (local address 127.9.238.254:38215)
0926 16:44:51.029480 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:51.029498 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:51.030923 (+ 1425us) server_negotiation.cc:407] Connection header received
0926 16:44:51.033073 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:51.033078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:51.033172 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:51.033306 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:51.035294 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.036444 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.038566 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.039016 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.039183 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:51.040833 (+ 1650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:51.040838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:51.040841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:44:51.040860 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:51.040898 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:44:51.040911 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:51.041010 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:51.041248 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 16:44:51.041354 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":250,"threads_started":1}
I20260926 16:44:51.042842 19731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:51.029551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:38215 (local address 127.9.238.193:36951)
0926 16:44:51.030723 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:51.030778 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:51.032708 (+ 1930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:51.033675 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:51.033687 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:51.033870 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:51.035007 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:51.035045 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.036700 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.036706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:51.038380 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:51.038399 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.039362 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.039369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:51.039668 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:51.040472 (+ 804us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:44:51.040474 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:51.040520 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:44:51.040614 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:51.041011 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:51.041024 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:51.041034 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 16:44:51.041240 (+ 206us) client_negotiation.cc:240] Negotiation successful
0926 16:44:51.041678 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":365,"threads_started":1}
I20260926 16:44:51.044739 19728 heartbeater.cc:347] Connected to a master server at 127.9.238.254:38215
I20260926 16:44:51.045075 19728 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:51.045883 19728 heartbeater.cc:511] Master 127.9.238.254:38215 requested a full tablet report, sending...
I20260926 16:44:51.048382 19542 ts_manager.cc:194] Registered new tserver with Master: 6623e3b6dc9a485ebe6287d3a95e7142 (127.9.238.193:42993)
I20260926 16:44:51.049890 19542 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.238.193:36951
W20260926 16:44:51.406209 19732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:51.406601 19732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:51.406740 19732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:51.414326 19732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:51.414503 19732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:51.414680 19732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:44:51.424454 19732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:38275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:38215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:51.425885 19732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:51.427863 19732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:51.443197 19740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:51.443262 19737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:51.443514 19738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:51.444522 19732 server_base.cc:1034] running on GCE node
I20260926 16:44:51.445255 19732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:51.446494 19732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:51.447718 19732 hybrid_clock.cc:648] HybridClock initialized: now 1790441091447638 us; error 76 us; skew 500 ppm
I20260926 16:44:51.448144 19732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:44:51.451790 19732 webserver.cc:460] Webserver started at http://127.9.238.194:44879/ using document root <none> and password file <none>
I20260926 16:44:51.452899 19732 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:51.453035 19732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:51.453504 19732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:51.457674 19732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/data/instance:
uuid: "b38b0d0485bc4edea72e087a15a445f0"
format_stamp: "Formatted at 2026-09-26 16:44:51 on dist-test-slave-wt2j"
server_key: "e4ccbb7c2eacc6433b86bd40080f7c82"
server_key_iv: "dc0bf806a3655f086a84387c0fdb9e43"
server_key_version: "encryptionkey@0"
I20260926 16:44:51.459033 19732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance:
uuid: "b38b0d0485bc4edea72e087a15a445f0"
format_stamp: "Formatted at 2026-09-26 16:44:51 on dist-test-slave-wt2j"
server_key: "e4ccbb7c2eacc6433b86bd40080f7c82"
server_key_iv: "dc0bf806a3655f086a84387c0fdb9e43"
server_key_version: "encryptionkey@0"
I20260926 16:44:51.466933 19732 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260926 16:44:51.472563 19746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:51.474747 19732 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260926 16:44:51.475018 19732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "b38b0d0485bc4edea72e087a15a445f0"
format_stamp: "Formatted at 2026-09-26 16:44:51 on dist-test-slave-wt2j"
server_key: "e4ccbb7c2eacc6433b86bd40080f7c82"
server_key_iv: "dc0bf806a3655f086a84387c0fdb9e43"
server_key_version: "encryptionkey@0"
I20260926 16:44:51.475337 19732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:51.495910 19732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:51.497140 19732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:51.497542 19732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:51.498905 19732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:51.501081 19732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:51.501194 19732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:51.501386 19732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:51.501502 19732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:51.545694 19732 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:38587
I20260926 16:44:51.545764 19858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:38587 every 8 connection(s)
I20260926 16:44:51.547535 19732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:44:51.550297 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 19732
I20260926 16:44:51.550480 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance
I20260926 16:44:51.551112 10171 external_mini_cluster.cc:1428] Setting key cee691560486ec6911ac976a222556a8
I20260926 16:44:51.554937 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:0
--local_ip_for_outbound_sockets=127.9.238.195
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:38215
--builtin_ntp_servers=127.9.238.212:38275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 16:44:51.563876 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:51.551351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:55331 (local address 127.9.238.254:38215)
0926 16:44:51.551965 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:51.551970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:51.553473 (+ 1503us) server_negotiation.cc:407] Connection header received
0926 16:44:51.555914 (+ 2441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:51.555920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:51.556028 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:51.556179 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:51.557639 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.558476 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.560379 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.560824 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.560975 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:51.562806 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:51.562812 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:51.562815 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:44:51.562841 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:51.562889 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:44:51.562904 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:51.563010 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:51.563278 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 16:44:51.563423 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":343,"threads_started":1}
I20260926 16:44:51.565284 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:51.552029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:38215 (local address 127.9.238.194:55331)
0926 16:44:51.553257 (+ 1228us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:51.553307 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:51.555502 (+ 2195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:51.556507 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:51.556517 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:51.556636 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:51.557433 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:51.557456 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.558674 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.558679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:51.560211 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:51.560224 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:51.561092 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:51.561099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:51.561429 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:51.562360 (+ 931us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:44:51.562363 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:51.562422 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:44:51.562533 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:51.563009 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:51.563022 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:51.563034 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 16:44:51.563274 (+ 240us) client_negotiation.cc:240] Negotiation successful
0926 16:44:51.563885 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":408,"threads_started":1}
I20260926 16:44:51.567440 19859 heartbeater.cc:347] Connected to a master server at 127.9.238.254:38215
I20260926 16:44:51.567938 19859 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:51.569003 19859 heartbeater.cc:511] Master 127.9.238.254:38215 requested a full tablet report, sending...
I20260926 16:44:51.571164 19542 ts_manager.cc:194] Registered new tserver with Master: b38b0d0485bc4edea72e087a15a445f0 (127.9.238.194:38587)
I20260926 16:44:51.572726 19542 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.238.194:55331
W20260926 16:44:51.908876 19863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:51.909196 19863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:51.909279 19863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:51.916697 19863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:51.916824 19863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:51.916932 19863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:44:51.926473 19863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:38275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:38215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:51.927835 19863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:51.929567 19863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:51.944231 19868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:51.944362 19869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:51.944638 19871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:51.945546 19863 server_base.cc:1034] running on GCE node
I20260926 16:44:51.946235 19863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:51.947347 19863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:51.948701 19863 hybrid_clock.cc:648] HybridClock initialized: now 1790441091948633 us; error 42 us; skew 500 ppm
I20260926 16:44:51.949071 19863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:44:51.951808 19863 webserver.cc:460] Webserver started at http://127.9.238.195:33351/ using document root <none> and password file <none>
I20260926 16:44:51.952589 19863 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:51.952692 19863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:51.953063 19863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:51.956157 19863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/data/instance:
uuid: "bb0f834fb97c41eca25d0d557739994e"
format_stamp: "Formatted at 2026-09-26 16:44:51 on dist-test-slave-wt2j"
server_key: "00e6e79ea893a170de5bcbf164418761"
server_key_iv: "d50f30ef196d8288e6b23c475ae43b4f"
server_key_version: "encryptionkey@0"
I20260926 16:44:51.957006 19863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance:
uuid: "bb0f834fb97c41eca25d0d557739994e"
format_stamp: "Formatted at 2026-09-26 16:44:51 on dist-test-slave-wt2j"
server_key: "00e6e79ea893a170de5bcbf164418761"
server_key_iv: "d50f30ef196d8288e6b23c475ae43b4f"
server_key_version: "encryptionkey@0"
I20260926 16:44:51.963168 19863 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260926 16:44:51.968010 19877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:51.969942 19863 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:44:51.970194 19863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "bb0f834fb97c41eca25d0d557739994e"
format_stamp: "Formatted at 2026-09-26 16:44:51 on dist-test-slave-wt2j"
server_key: "00e6e79ea893a170de5bcbf164418761"
server_key_iv: "d50f30ef196d8288e6b23c475ae43b4f"
server_key_version: "encryptionkey@0"
I20260926 16:44:51.970438 19863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:51.992245 19863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:51.993191 19863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:51.993526 19863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:51.994670 19863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:51.996424 19863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:51.996517 19863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:51.996661 19863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:51.996742 19863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:52.042929 19863 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:38237
I20260926 16:44:52.043180 19989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:38237 every 8 connection(s)
I20260926 16:44:52.044313 19863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:44:52.053068 19728 heartbeater.cc:503] Master 127.9.238.254:38215 was elected leader, sending a full tablet report...
I20260926 16:44:52.054121 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 19863
I20260926 16:44:52.054320 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance
I20260926 16:44:52.054890 10171 external_mini_cluster.cc:1428] Setting key 2acccdb482b98b5af471e1db4e6bad4b
I20260926 16:44:52.060118 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:52.047773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:54559 (local address 127.9.238.254:38215)
0926 16:44:52.047970 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:52.047975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:52.049790 (+ 1815us) server_negotiation.cc:407] Connection header received
0926 16:44:52.051789 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:52.051794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:52.051902 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:52.052028 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:52.053176 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.054227 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.056628 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.057423 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.057549 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:52.059165 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:52.059170 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:52.059174 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:44:52.059194 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:52.059235 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:44:52.059248 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:52.059336 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:52.059704 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 16:44:52.059816 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 16:44:52.061149 19992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:52.048381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:38215 (local address 127.9.238.195:54559)
0926 16:44:52.049586 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:52.049638 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:52.051396 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:52.052283 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:52.052292 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:52.052388 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:52.053005 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:52.053023 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.054444 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.054450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:52.056428 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:52.056444 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.057679 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.057687 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:52.058001 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:52.058816 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:44:52.058819 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:52.058859 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:44:52.058961 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:52.059410 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:52.059419 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:52.059425 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 16:44:52.059686 (+ 261us) client_negotiation.cc:240] Negotiation successful
0926 16:44:52.060004 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":359,"threads_started":1}
I20260926 16:44:52.063126 19990 heartbeater.cc:347] Connected to a master server at 127.9.238.254:38215
I20260926 16:44:52.063501 19990 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:52.064400 19990 heartbeater.cc:511] Master 127.9.238.254:38215 requested a full tablet report, sending...
I20260926 16:44:52.066013 19542 ts_manager.cc:194] Registered new tserver with Master: bb0f834fb97c41eca25d0d557739994e (127.9.238.195:38237)
I20260926 16:44:52.067070 19542 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.238.195:54559
I20260926 16:44:52.071219 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 16:44:52.078770 19861 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:52.076832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54206 (local address 127.9.238.254:38215)
0926 16:44:52.077128 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:52.077136 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:52.077606 (+ 470us) server_negotiation.cc:407] Connection header received
0926 16:44:52.077776 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:52.077781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:52.077861 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:52.077994 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:52.078472 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54206: BlockingRecv error: recv got EOF from 127.0.0.1:54206 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 16:44:52.079746 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 19601
I20260926 16:44:52.089821 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:52.080983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50894 (local address 127.9.238.193:42993)
0926 16:44:52.081742 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:52.081750 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:52.081785 (+ 35us) server_negotiation.cc:407] Connection header received
0926 16:44:52.081861 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:52.081870 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:52.082160 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:52.082362 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:52.082856 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.084389 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.085798 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.086541 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.086668 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:52.087878 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:52.087889 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:52.087908 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:44:52.088688 (+ 780us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:52.088764 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:44:52.088788 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:52.088902 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:52.089211 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 16:44:52.089381 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":393,"threads_started":1}
I20260926 16:44:52.091718 19723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:52.186708 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 19601
I20260926 16:44:52.204057 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 19732
I20260926 16:44:52.212515 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:52.205238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48388 (local address 127.9.238.194:38587)
0926 16:44:52.205860 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:52.205868 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:52.205897 (+ 29us) server_negotiation.cc:407] Connection header received
0926 16:44:52.205971 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:52.205978 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:52.206218 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:52.206408 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:52.206823 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.207911 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.209146 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.209704 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.209838 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:52.210945 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:52.210953 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:52.210968 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:44:52.211649 (+ 681us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:52.211708 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:44:52.211728 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:52.211851 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:52.211998 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 16:44:52.212167 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":300,"threads_started":1}
I20260926 16:44:52.214165 19854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:52.309629 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 19732
I20260926 16:44:52.328444 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 19863
I20260926 16:44:52.339210 20003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:52.329784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44016 (local address 127.9.238.195:38237)
0926 16:44:52.330580 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:52.330589 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:52.330622 (+ 33us) server_negotiation.cc:407] Connection header received
0926 16:44:52.330684 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:52.330691 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:52.330920 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:52.331138 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:52.331869 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.333101 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.334674 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:52.335456 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:52.335763 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:52.337511 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:52.337520 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:52.337535 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:44:52.338249 (+ 714us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:52.338309 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:44:52.338330 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:52.338447 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:52.338731 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 16:44:52.338853 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":413,"threads_started":1}
I20260926 16:44:52.341254 19985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:52.449858 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 19863
I20260926 16:44:52.470273 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 19511
I20260926 16:44:52.471868 19572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:52.579701 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 19511
2026-09-26T16:44:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2759 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T16:44:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:44:52Z Disabled control of system clock
I20260926 16:44:52.666443 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:36305
--webserver_interface=127.9.238.254
--webserver_port=0
--builtin_ntp_servers=127.9.238.212:40375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:36305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 16:44:53.003777 20009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:53.004206 20009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:53.004387 20009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:53.011871 20009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:44:53.011991 20009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:53.012036 20009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:53.012138 20009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:44:53.012213 20009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:44:53.021751 20009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:40375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:36305
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:36305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:53.023270 20009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:53.025189 20009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:53.039443 20014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:53.039695 20017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:53.039397 20015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:53.040133 20009 server_base.cc:1034] running on GCE node
I20260926 16:44:53.041222 20009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:53.042961 20009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:53.044317 20009 hybrid_clock.cc:648] HybridClock initialized: now 1790441093044264 us; error 71 us; skew 500 ppm
I20260926 16:44:53.044705 20009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:44:53.047441 20009 webserver.cc:460] Webserver started at http://127.9.238.254:36761/ using document root <none> and password file <none>
I20260926 16:44:53.048226 20009 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:53.048357 20009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:53.048708 20009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:53.051760 20009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "615d226c2d75480ebd45b41c0bd3c89e"
format_stamp: "Formatted at 2026-09-26 16:44:53 on dist-test-slave-wt2j"
server_key: "7e2a85faf43d34eacd6b8be0934cfbb8"
server_key_iv: "b41ebafb5bc0a09f6b81b24a6217c650"
server_key_version: "encryptionkey@0"
I20260926 16:44:53.052569 20009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "615d226c2d75480ebd45b41c0bd3c89e"
format_stamp: "Formatted at 2026-09-26 16:44:53 on dist-test-slave-wt2j"
server_key: "7e2a85faf43d34eacd6b8be0934cfbb8"
server_key_iv: "b41ebafb5bc0a09f6b81b24a6217c650"
server_key_version: "encryptionkey@0"
I20260926 16:44:53.058182 20009 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260926 16:44:53.062433 20023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:53.064301 20009 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:44:53.064550 20009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "615d226c2d75480ebd45b41c0bd3c89e"
format_stamp: "Formatted at 2026-09-26 16:44:53 on dist-test-slave-wt2j"
server_key: "7e2a85faf43d34eacd6b8be0934cfbb8"
server_key_iv: "b41ebafb5bc0a09f6b81b24a6217c650"
server_key_version: "encryptionkey@0"
I20260926 16:44:53.064810 20009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:53.071105 20009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:53.072039 20009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:53.072358 20009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:53.094323 20009 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:36305
I20260926 16:44:53.094429 20074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:36305 every 8 connection(s)
I20260926 16:44:53.095990 20009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:44:53.101282 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 20009
I20260926 16:44:53.101522 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:44:53.101696 20075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:53.102329 10171 external_mini_cluster.cc:1428] Setting key 5400afd0de171ec0e741a1cab966d192
I20260926 16:44:53.116113 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:53.105129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53210 (local address 127.9.238.254:36305)
0926 16:44:53.106676 (+ 1547us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:53.106696 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:53.106769 (+ 73us) server_negotiation.cc:407] Connection header received
0926 16:44:53.110345 (+ 3576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:53.110400 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:53.111302 (+ 902us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:53.112022 (+ 720us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:53.113407 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:53.113418 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:53.113440 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:44:53.113464 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:53.113567 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:44:53.113604 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:53.113921 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:53.114083 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 16:44:53.114476 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":479,"threads_started":1}
I20260926 16:44:53.120545 20075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 615d226c2d75480ebd45b41c0bd3c89e: Bootstrap starting.
I20260926 16:44:53.125101 20075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 615d226c2d75480ebd45b41c0bd3c89e: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:53.126628 20075 log.cc:824] T 00000000000000000000000000000000 P 615d226c2d75480ebd45b41c0bd3c89e: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:53.130321 20075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 615d226c2d75480ebd45b41c0bd3c89e: No bootstrap required, opened a new log
I20260926 16:44:53.139806 20075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 615d226c2d75480ebd45b41c0bd3c89e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615d226c2d75480ebd45b41c0bd3c89e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 36305 } }
I20260926 16:44:53.140333 20075 raft_consensus.cc:374] T 00000000000000000000000000000000 P 615d226c2d75480ebd45b41c0bd3c89e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:53.140424 20075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 615d226c2d75480ebd45b41c0bd3c89e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615d226c2d75480ebd45b41c0bd3c89e, State: Initialized, Role: FOLLOWER
I20260926 16:44:53.141149 20075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 615d226c2d75480ebd45b41c0bd3c89e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615d226c2d75480ebd45b41c0bd3c89e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 36305 } }
I20260926 16:44:53.141402 20075 raft_consensus.cc:388] T 00000000000000000000000000000000 P 615d226c2d75480ebd45b41c0bd3c89e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:44:53.141528 20075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 615d226c2d75480ebd45b41c0bd3c89e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:44:53.141687 20075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 615d226c2d75480ebd45b41c0bd3c89e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:53.144810 20075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 615d226c2d75480ebd45b41c0bd3c89e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615d226c2d75480ebd45b41c0bd3c89e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 36305 } }
I20260926 16:44:53.145593 20075 leader_election.cc:302] T 00000000000000000000000000000000 P 615d226c2d75480ebd45b41c0bd3c89e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 615d226c2d75480ebd45b41c0bd3c89e; no voters:
I20260926 16:44:53.146363 20075 leader_election.cc:288] T 00000000000000000000000000000000 P 615d226c2d75480ebd45b41c0bd3c89e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:44:53.146605 20080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 615d226c2d75480ebd45b41c0bd3c89e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:44:53.147612 20080 raft_consensus.cc:686] T 00000000000000000000000000000000 P 615d226c2d75480ebd45b41c0bd3c89e [term 1 LEADER]: Becoming Leader. State: Replica: 615d226c2d75480ebd45b41c0bd3c89e, State: Running, Role: LEADER
I20260926 16:44:53.148423 20080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 615d226c2d75480ebd45b41c0bd3c89e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615d226c2d75480ebd45b41c0bd3c89e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 36305 } }
I20260926 16:44:53.149374 20075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 615d226c2d75480ebd45b41c0bd3c89e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:44:53.151746 20081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 615d226c2d75480ebd45b41c0bd3c89e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "615d226c2d75480ebd45b41c0bd3c89e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615d226c2d75480ebd45b41c0bd3c89e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 36305 } } }
I20260926 16:44:53.152207 20081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 615d226c2d75480ebd45b41c0bd3c89e [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:53.152774 20082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 615d226c2d75480ebd45b41c0bd3c89e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 615d226c2d75480ebd45b41c0bd3c89e. Latest consensus state: current_term: 1 leader_uuid: "615d226c2d75480ebd45b41c0bd3c89e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615d226c2d75480ebd45b41c0bd3c89e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 36305 } } }
I20260926 16:44:53.153260 20082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 615d226c2d75480ebd45b41c0bd3c89e [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:53.155453 20087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:44:53.161073 20087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:44:53.172857 20087 catalog_manager.cc:1347] Generated new cluster ID: 533fca3fa66f4890a5c11449a3bb6074
I20260926 16:44:53.173044 20087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:44:53.194629 20087 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:44:53.195137 20087 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:44:53.206800 20087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 615d226c2d75480ebd45b41c0bd3c89e: Generated new TSK 0
I20260926 16:44:53.207877 20087 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:44:53.220873 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:0
--local_ip_for_outbound_sockets=127.9.238.193
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:36305
--builtin_ntp_servers=127.9.238.212:40375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 16:44:53.650444 20099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:53.650924 20099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:53.651093 20099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:53.659653 20099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:53.659780 20099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:53.659972 20099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:44:53.670715 20099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:40375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:36305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:53.672629 20099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:53.674810 20099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:53.690933 20104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:53.691076 20105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:53.691080 20107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:53.692421 20099 server_base.cc:1034] running on GCE node
I20260926 16:44:53.693280 20099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:53.694561 20099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:53.695859 20099 hybrid_clock.cc:648] HybridClock initialized: now 1790441093695796 us; error 51 us; skew 500 ppm
I20260926 16:44:53.696493 20099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:44:53.700279 20099 webserver.cc:460] Webserver started at http://127.9.238.193:40915/ using document root <none> and password file <none>
I20260926 16:44:53.701647 20099 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:53.701826 20099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:53.702267 20099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:53.706113 20099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/data/instance:
uuid: "b713589a19a442a1835d3dd324400329"
format_stamp: "Formatted at 2026-09-26 16:44:53 on dist-test-slave-wt2j"
server_key: "d7c3f480c6dc92c1c50de5a089a2f175"
server_key_iv: "7e7b06e22bdfddbf18c2d6a0e36f3bd3"
server_key_version: "encryptionkey@0"
I20260926 16:44:53.707319 20099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance:
uuid: "b713589a19a442a1835d3dd324400329"
format_stamp: "Formatted at 2026-09-26 16:44:53 on dist-test-slave-wt2j"
server_key: "d7c3f480c6dc92c1c50de5a089a2f175"
server_key_iv: "7e7b06e22bdfddbf18c2d6a0e36f3bd3"
server_key_version: "encryptionkey@0"
I20260926 16:44:53.714291 20099 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 16:44:53.719141 20113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:53.721068 20099 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:44:53.721372 20099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "b713589a19a442a1835d3dd324400329"
format_stamp: "Formatted at 2026-09-26 16:44:53 on dist-test-slave-wt2j"
server_key: "d7c3f480c6dc92c1c50de5a089a2f175"
server_key_iv: "7e7b06e22bdfddbf18c2d6a0e36f3bd3"
server_key_version: "encryptionkey@0"
I20260926 16:44:53.721700 20099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:53.729311 20099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:53.730460 20099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:53.730801 20099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:53.732071 20099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:53.733899 20099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:53.734035 20099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:53.734184 20099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:53.734288 20099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:53.775404 20099 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:34793
I20260926 16:44:53.775465 20225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:34793 every 8 connection(s)
I20260926 16:44:53.777115 20099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:44:53.785252 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 20099
I20260926 16:44:53.785552 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance
I20260926 16:44:53.786471 10171 external_mini_cluster.cc:1428] Setting key fde9deaaecf6b8ebef27cf8aa388db5f
I20260926 16:44:53.786933 20228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:53.780226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:37753 (local address 127.9.238.254:36305)
0926 16:44:53.780718 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:53.780722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:53.782305 (+ 1583us) server_negotiation.cc:407] Connection header received
0926 16:44:53.784221 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:53.784226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:53.784312 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:53.784432 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:53.786134 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:53.786138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:53.786141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:44:53.786156 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:53.786194 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:44:53.786206 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:53.786342 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:53.786549 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 16:44:53.786621 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":293,"threads_started":1}
I20260926 16:44:53.788130 20229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:53.780910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:36305 (local address 127.9.238.193:37753)
0926 16:44:53.782123 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:53.782168 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:53.783929 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:53.784712 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:53.784721 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:53.784821 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:53.785760 (+ 939us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:44:53.785763 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:53.785802 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:44:53.785893 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:53.786321 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:53.786334 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:53.786343 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 16:44:53.786655 (+ 312us) client_negotiation.cc:240] Negotiation successful
0926 16:44:53.786939 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":351,"threads_started":1}
I20260926 16:44:53.790122 20226 heartbeater.cc:347] Connected to a master server at 127.9.238.254:36305
I20260926 16:44:53.790499 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:0
--local_ip_for_outbound_sockets=127.9.238.194
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:36305
--builtin_ntp_servers=127.9.238.212:40375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 16:44:53.790616 20226 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:53.791652 20226 heartbeater.cc:511] Master 127.9.238.254:36305 requested a full tablet report, sending...
I20260926 16:44:53.794986 20040 ts_manager.cc:194] Registered new tserver with Master: b713589a19a442a1835d3dd324400329 (127.9.238.193:34793)
W20260926 16:44:54.177695 20230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:54.178004 20230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:54.178151 20230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:54.185694 20230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:54.185817 20230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:54.186002 20230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:44:54.195323 20230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:40375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:36305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:54.196764 20230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:54.198561 20230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:54.212960 20235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:54.213042 20238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:54.213023 20236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:54.214285 20230 server_base.cc:1034] running on GCE node
I20260926 16:44:54.215112 20230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:54.216481 20230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:54.217754 20230 hybrid_clock.cc:648] HybridClock initialized: now 1790441094217682 us; error 59 us; skew 500 ppm
I20260926 16:44:54.218165 20230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:44:54.221248 20230 webserver.cc:460] Webserver started at http://127.9.238.194:42953/ using document root <none> and password file <none>
I20260926 16:44:54.222173 20230 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:54.222323 20230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:54.222714 20230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:54.226485 20230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/data/instance:
uuid: "3941b122873641228133cad64c4aab2b"
format_stamp: "Formatted at 2026-09-26 16:44:54 on dist-test-slave-wt2j"
server_key: "ba5eb0b8beb6fd8f7886ed85bcf228db"
server_key_iv: "93e4991fabd894eef6f61b51d2354778"
server_key_version: "encryptionkey@0"
I20260926 16:44:54.227602 20230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance:
uuid: "3941b122873641228133cad64c4aab2b"
format_stamp: "Formatted at 2026-09-26 16:44:54 on dist-test-slave-wt2j"
server_key: "ba5eb0b8beb6fd8f7886ed85bcf228db"
server_key_iv: "93e4991fabd894eef6f61b51d2354778"
server_key_version: "encryptionkey@0"
I20260926 16:44:54.234571 20230 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20260926 16:44:54.239419 20244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:54.241694 20230 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:44:54.242028 20230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "3941b122873641228133cad64c4aab2b"
format_stamp: "Formatted at 2026-09-26 16:44:54 on dist-test-slave-wt2j"
server_key: "ba5eb0b8beb6fd8f7886ed85bcf228db"
server_key_iv: "93e4991fabd894eef6f61b51d2354778"
server_key_version: "encryptionkey@0"
I20260926 16:44:54.242359 20230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:54.250226 20230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:54.251362 20230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:54.251796 20230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:54.252969 20230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:54.254791 20230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:54.254892 20230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:54.255082 20230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:54.255170 20230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:54.296895 20230 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:38461
I20260926 16:44:54.297040 20356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:38461 every 8 connection(s)
I20260926 16:44:54.298303 20230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:44:54.306149 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 20230
I20260926 16:44:54.306418 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance
I20260926 16:44:54.307047 10171 external_mini_cluster.cc:1428] Setting key 90749a92949cd7a552acc7af96d802f1
I20260926 16:44:54.309137 20359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:54.301892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:55493 (local address 127.9.238.254:36305)
0926 16:44:54.302525 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:54.302530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:54.303847 (+ 1317us) server_negotiation.cc:407] Connection header received
0926 16:44:54.306024 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:54.306030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:54.306112 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:54.306240 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:54.308155 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:54.308163 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:54.308167 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:44:54.308192 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:54.308249 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:44:54.308272 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:54.308412 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:54.308723 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 16:44:54.308791 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":408,"threads_started":1}
I20260926 16:44:54.310546 20360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:54.302419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:36305 (local address 127.9.238.194:55493)
0926 16:44:54.303650 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:54.303698 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:54.305558 (+ 1860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:54.306623 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:54.306638 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:54.306796 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:54.307736 (+ 940us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:44:54.307739 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:54.307805 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:44:54.307932 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:54.308432 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:54.308443 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:54.308450 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 16:44:54.308757 (+ 307us) client_negotiation.cc:240] Negotiation successful
0926 16:44:54.309024 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":321,"threads_started":1}
I20260926 16:44:54.311089 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:0
--local_ip_for_outbound_sockets=127.9.238.195
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:36305
--builtin_ntp_servers=127.9.238.212:40375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 16:44:54.312433 20357 heartbeater.cc:347] Connected to a master server at 127.9.238.254:36305
I20260926 16:44:54.312851 20357 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:54.313581 20357 heartbeater.cc:511] Master 127.9.238.254:36305 requested a full tablet report, sending...
I20260926 16:44:54.315191 20040 ts_manager.cc:194] Registered new tserver with Master: 3941b122873641228133cad64c4aab2b (127.9.238.194:38461)
W20260926 16:44:54.724627 20361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:54.725062 20361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:54.725234 20361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:54.733919 20361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:54.734067 20361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:54.734275 20361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:44:54.745048 20361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:40375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:36305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:54.746845 20361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:54.749171 20361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:54.765475 20367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:54.765540 20369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:54.765413 20366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:54.766530 20361 server_base.cc:1034] running on GCE node
I20260926 16:44:54.767299 20361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:54.768627 20361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:54.769884 20361 hybrid_clock.cc:648] HybridClock initialized: now 1790441094769829 us; error 42 us; skew 500 ppm
I20260926 16:44:54.770270 20361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:44:54.773048 20361 webserver.cc:460] Webserver started at http://127.9.238.195:45473/ using document root <none> and password file <none>
I20260926 16:44:54.773941 20361 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:54.774085 20361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:54.774520 20361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:54.779279 20361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/data/instance:
uuid: "f35da6cbf4c441b888d34f25193ea969"
format_stamp: "Formatted at 2026-09-26 16:44:54 on dist-test-slave-wt2j"
server_key: "4a88c9a71d0d8b4af8543de9ff3f6cc3"
server_key_iv: "d7bb6b3a93c9744502735abe57a55d01"
server_key_version: "encryptionkey@0"
I20260926 16:44:54.780534 20361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance:
uuid: "f35da6cbf4c441b888d34f25193ea969"
format_stamp: "Formatted at 2026-09-26 16:44:54 on dist-test-slave-wt2j"
server_key: "4a88c9a71d0d8b4af8543de9ff3f6cc3"
server_key_iv: "d7bb6b3a93c9744502735abe57a55d01"
server_key_version: "encryptionkey@0"
I20260926 16:44:54.787493 20361 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 16:44:54.791913 20375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:54.793805 20361 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:44:54.794065 20361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "f35da6cbf4c441b888d34f25193ea969"
format_stamp: "Formatted at 2026-09-26 16:44:54 on dist-test-slave-wt2j"
server_key: "4a88c9a71d0d8b4af8543de9ff3f6cc3"
server_key_iv: "d7bb6b3a93c9744502735abe57a55d01"
server_key_version: "encryptionkey@0"
I20260926 16:44:54.794359 20361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:54.799801 20226 heartbeater.cc:503] Master 127.9.238.254:36305 was elected leader, sending a full tablet report...
I20260926 16:44:54.801389 20361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:54.802433 20361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:54.802690 20361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:54.803845 20361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:54.805533 20361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:54.805629 20361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:54.805729 20361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:54.805792 20361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:54.849188 20361 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:40941
I20260926 16:44:54.849254 20487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:40941 every 8 connection(s)
I20260926 16:44:54.851081 20361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:44:54.852249 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 20361
I20260926 16:44:54.852460 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance
I20260926 16:44:54.853080 10171 external_mini_cluster.cc:1428] Setting key 60a2e38d3727a160d27e17c3d51546e9
I20260926 16:44:54.861893 20490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:54.854878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:60321 (local address 127.9.238.254:36305)
0926 16:44:54.855506 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:54.855513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:54.857562 (+ 2049us) server_negotiation.cc:407] Connection header received
0926 16:44:54.859526 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:54.859530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:54.859652 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:54.859755 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:54.861202 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:54.861206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:54.861209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:44:54.861224 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:54.861266 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:44:54.861278 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:54.861377 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:54.861610 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 16:44:54.861659 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":338,"threads_started":1}
I20260926 16:44:54.863030 20491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:54.855984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:36305 (local address 127.9.238.195:60321)
0926 16:44:54.857361 (+ 1377us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:54.857432 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:54.859229 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:54.859998 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:54.860007 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:54.860115 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:54.860900 (+ 785us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:44:54.860904 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:54.860948 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:44:54.861042 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:54.861372 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:54.861380 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:54.861385 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 16:44:54.861652 (+ 267us) client_negotiation.cc:240] Negotiation successful
0926 16:44:54.861882 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":429,"threads_started":1}
I20260926 16:44:54.864894 20488 heartbeater.cc:347] Connected to a master server at 127.9.238.254:36305
I20260926 16:44:54.865337 20488 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:54.866109 20488 heartbeater.cc:511] Master 127.9.238.254:36305 requested a full tablet report, sending...
I20260926 16:44:54.868067 20040 ts_manager.cc:194] Registered new tserver with Master: f35da6cbf4c441b888d34f25193ea969 (127.9.238.195:40941)
I20260926 16:44:54.870729 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 16:44:54.878226 20490 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:44:54.876266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53214 (local address 127.9.238.254:36305)
0926 16:44:54.876518 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:54.876527 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:54.877035 (+ 508us) server_negotiation.cc:407] Connection header received
0926 16:44:54.877286 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:54.877290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:54.877351 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:54.877465 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:54.877873 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53214: BlockingRecv error: recv got EOF from 127.0.0.1:53214 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 16:44:54.879174 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 20099
I20260926 16:44:54.884176 20498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:54.881016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41134 (local address 127.9.238.193:34793)
0926 16:44:54.881756 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:54.881765 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:54.881799 (+ 34us) server_negotiation.cc:407] Connection header received
0926 16:44:54.881886 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:54.881893 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:54.882165 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:54.882351 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:54.883330 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:54.883338 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:54.883351 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:44:54.883363 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:54.883410 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:44:54.883425 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:54.883664 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:54.883759 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 16:44:54.883856 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":399,"threads_started":1}
I20260926 16:44:54.885722 20221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:54.976747 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 20099
I20260926 16:44:54.994486 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 20230
I20260926 16:44:54.999009 20500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:54.995738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42058 (local address 127.9.238.194:38461)
0926 16:44:54.996391 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:54.996401 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:54.996437 (+ 36us) server_negotiation.cc:407] Connection header received
0926 16:44:54.996524 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:54.996533 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:54.996838 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:54.997009 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:54.997999 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:54.998008 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:54.998022 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:44:54.998035 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:54.998087 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:44:54.998108 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:54.998231 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:54.998438 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 16:44:54.998590 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":256,"threads_started":1}
I20260926 16:44:55.000859 20352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:55.087491 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 20230
I20260926 16:44:55.108800 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 20361
I20260926 16:44:55.113880 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:55.110258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46620 (local address 127.9.238.195:40941)
0926 16:44:55.111189 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:55.111198 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:55.111239 (+ 41us) server_negotiation.cc:407] Connection header received
0926 16:44:55.111323 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:55.111331 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:55.111691 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:55.111865 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:55.112982 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:55.112990 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:55.113005 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:44:55.113023 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:55.113073 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:44:55.113088 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:55.113256 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:55.113319 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 16:44:55.113464 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":456,"threads_started":1}
I20260926 16:44:55.115410 20483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:55.201067 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 20361
I20260926 16:44:55.219887 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 20009
I20260926 16:44:55.221199 20070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:55.325492 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 20009
2026-09-26T16:44:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2746 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:44:55 dist-test-slave-wt2j krb5kdc[20507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:44:55 dist-test-slave-wt2j krb5kdc[20507](info): set up 2 sockets
Sep 26 16:44:55 dist-test-slave-wt2j krb5kdc[20507](info): commencing operation
krb5kdc: starting...
W20260926 16:44:56.485755 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.000s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:44:56 dist-test-slave-wt2j krb5kdc[20507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:44:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:44:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:44:56.771667 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:44201
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:44201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:57.139444 20520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:57.139861 20520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:57.140043 20520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:57.147399 20520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:44:57.147626 20520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:57.147742 20520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:57.147850 20520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:44:57.147919 20520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:44:57.158054 20520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:44201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:44201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:57.159610 20520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:57.161000 20520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:57.176584 20526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:57.176605 20525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:57.176600 20528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:57.177680 20520 server_base.cc:1034] running on GCE node
I20260926 16:44:57.178813 20520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:57.180969 20520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:57.182382 20520 hybrid_clock.cc:648] HybridClock initialized: now 1790441097182216 us; error 52 us; skew 500 ppm
I20260926 16:44:57.182857 20520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:57 dist-test-slave-wt2j krb5kdc[20507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441097, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:57.192181 20520 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:44:57.194458 20520 webserver.cc:460] Webserver started at http://127.9.238.254:35885/ using document root <none> and password file <none>
I20260926 16:44:57.195511 20520 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:57.195987 20520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:57.196455 20520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:57.200639 20520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "6bad906ec6df4e37b3e86eb02a1dc5bb"
format_stamp: "Formatted at 2026-09-26 16:44:57 on dist-test-slave-wt2j"
server_key: "87802b78d5fb1211e6c88c1f0f78306b"
server_key_iv: "76b5f5b0ff4e14f7fe447a927bacbe97"
server_key_version: "encryptionkey@0"
I20260926 16:44:57.201655 20520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "6bad906ec6df4e37b3e86eb02a1dc5bb"
format_stamp: "Formatted at 2026-09-26 16:44:57 on dist-test-slave-wt2j"
server_key: "87802b78d5fb1211e6c88c1f0f78306b"
server_key_iv: "76b5f5b0ff4e14f7fe447a927bacbe97"
server_key_version: "encryptionkey@0"
I20260926 16:44:57.209057 20520 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260926 16:44:57.213867 20535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:57.215857 20520 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 16:44:57.216099 20520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "6bad906ec6df4e37b3e86eb02a1dc5bb"
format_stamp: "Formatted at 2026-09-26 16:44:57 on dist-test-slave-wt2j"
server_key: "87802b78d5fb1211e6c88c1f0f78306b"
server_key_iv: "76b5f5b0ff4e14f7fe447a927bacbe97"
server_key_version: "encryptionkey@0"
I20260926 16:44:57.216434 20520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:57.238240 20520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:57.243927 20520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:57.244267 20520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:57.267498 20520 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:44201
I20260926 16:44:57.267572 20586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:44201 every 8 connection(s)
I20260926 16:44:57.269315 20520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:44:57.273469 20587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:57.274852 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 20520
I20260926 16:44:57.275112 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:44:57.275959 10171 external_mini_cluster.cc:1428] Setting key adaa0152ffd1383bcce2a63525521a41
I20260926 16:44:57.287261 20587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bad906ec6df4e37b3e86eb02a1dc5bb: Bootstrap starting.
I20260926 16:44:57.291148 20587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bad906ec6df4e37b3e86eb02a1dc5bb: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:57.292608 20587 log.cc:824] T 00000000000000000000000000000000 P 6bad906ec6df4e37b3e86eb02a1dc5bb: Log is configured to *not* fsync() on all Append() calls
Sep 26 16:44:57 dist-test-slave-wt2j krb5kdc[20507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:44:57.296425 20587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bad906ec6df4e37b3e86eb02a1dc5bb: No bootstrap required, opened a new log
I20260926 16:44:57.304394 20590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:57.278584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51006 (local address 127.9.238.254:44201)
0926 16:44:57.279718 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:57.279737 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:57.279805 (+ 68us) server_negotiation.cc:407] Connection header received
0926 16:44:57.281846 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:57.281878 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:57.282434 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:57.282858 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:57.284849 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:57.286535 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:57.288420 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:57.289370 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:57.289546 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:57.294587 (+ 5041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:57.294598 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:57.294619 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:57.294666 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:57.298756 (+ 4090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:57.299693 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:57.299700 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:57.299709 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:57.299911 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:57.300319 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:57.300328 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:57.300330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:57.301050 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:57.301453 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:57.301934 (+ 481us) server_negotiation.cc:299] Negotiation successful
0926 16:44:57.302483 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":799,"thread_start_us":516,"threads_started":1}
I20260926 16:44:57.305744 20587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bad906ec6df4e37b3e86eb02a1dc5bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bad906ec6df4e37b3e86eb02a1dc5bb" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44201 } }
I20260926 16:44:57.306185 20587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bad906ec6df4e37b3e86eb02a1dc5bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:57.306325 20587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bad906ec6df4e37b3e86eb02a1dc5bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bad906ec6df4e37b3e86eb02a1dc5bb, State: Initialized, Role: FOLLOWER
I20260926 16:44:57.306977 20587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bad906ec6df4e37b3e86eb02a1dc5bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bad906ec6df4e37b3e86eb02a1dc5bb" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44201 } }
I20260926 16:44:57.307228 20587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bad906ec6df4e37b3e86eb02a1dc5bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:44:57.307392 20587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bad906ec6df4e37b3e86eb02a1dc5bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:44:57.307639 20587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bad906ec6df4e37b3e86eb02a1dc5bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:57.310102 20587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bad906ec6df4e37b3e86eb02a1dc5bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bad906ec6df4e37b3e86eb02a1dc5bb" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44201 } }
I20260926 16:44:57.310679 20587 leader_election.cc:302] T 00000000000000000000000000000000 P 6bad906ec6df4e37b3e86eb02a1dc5bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bad906ec6df4e37b3e86eb02a1dc5bb; no voters:
I20260926 16:44:57.311398 20587 leader_election.cc:288] T 00000000000000000000000000000000 P 6bad906ec6df4e37b3e86eb02a1dc5bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:44:57.311764 20592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bad906ec6df4e37b3e86eb02a1dc5bb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:44:57.312815 20592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bad906ec6df4e37b3e86eb02a1dc5bb [term 1 LEADER]: Becoming Leader. State: Replica: 6bad906ec6df4e37b3e86eb02a1dc5bb, State: Running, Role: LEADER
I20260926 16:44:57.313686 20592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bad906ec6df4e37b3e86eb02a1dc5bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bad906ec6df4e37b3e86eb02a1dc5bb" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44201 } }
I20260926 16:44:57.313971 20587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bad906ec6df4e37b3e86eb02a1dc5bb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:44:57.317323 20593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bad906ec6df4e37b3e86eb02a1dc5bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bad906ec6df4e37b3e86eb02a1dc5bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bad906ec6df4e37b3e86eb02a1dc5bb" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44201 } } }
I20260926 16:44:57.317950 20594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bad906ec6df4e37b3e86eb02a1dc5bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bad906ec6df4e37b3e86eb02a1dc5bb. Latest consensus state: current_term: 1 leader_uuid: "6bad906ec6df4e37b3e86eb02a1dc5bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bad906ec6df4e37b3e86eb02a1dc5bb" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44201 } } }
I20260926 16:44:57.318554 20593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bad906ec6df4e37b3e86eb02a1dc5bb [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:57.318559 20594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bad906ec6df4e37b3e86eb02a1dc5bb [sys.catalog]: This master's current role is: LEADER
I20260926 16:44:57.321219 20599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:44:57.327162 20599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:44:57.338758 20599 catalog_manager.cc:1347] Generated new cluster ID: 09d86a124a9b4a289fd4cc8250299315
I20260926 16:44:57.338976 20599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:44:57.351626 20599 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:44:57.353912 20599 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:44:57.368342 20599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bad906ec6df4e37b3e86eb02a1dc5bb: Generated new TSK 0
I20260926 16:44:57.369494 20599 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
I20260926 16:44:57.448956 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:0
--local_ip_for_outbound_sockets=127.9.238.193
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44201
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:57.865226 20613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:57.865631 20613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:57.865803 20613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:57.874887 20613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:57.875051 20613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:57.875248 20613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:44:57.886965 20613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:57.888855 20613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:57.890534 20613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:57.907440 20618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:57.907481 20621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:57.907440 20619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:57.908527 20613 server_base.cc:1034] running on GCE node
I20260926 16:44:57.909376 20613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:57.910657 20613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:57.911914 20613 hybrid_clock.cc:648] HybridClock initialized: now 1790441097911797 us; error 70 us; skew 500 ppm
I20260926 16:44:57.912307 20613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:57 dist-test-slave-wt2j krb5kdc[20507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441097, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:57.916836 20613 init.cc:377] Logged in from keytab as kudu/127.9.238.193@KRBTEST.COM (short username kudu)
I20260926 16:44:57.918956 20613 webserver.cc:460] Webserver started at http://127.9.238.193:42983/ using document root <none> and password file <none>
I20260926 16:44:57.920042 20613 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:57.920202 20613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:57.920640 20613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:57.924957 20613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/data/instance:
uuid: "e35b57ae095742938433b3962e07d1b9"
format_stamp: "Formatted at 2026-09-26 16:44:57 on dist-test-slave-wt2j"
server_key: "8c367e4ce4aa1aac0302bf56e7fc8933"
server_key_iv: "b63e75c70ddd2dfca5a3da162caec950"
server_key_version: "encryptionkey@0"
I20260926 16:44:57.925974 20613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance:
uuid: "e35b57ae095742938433b3962e07d1b9"
format_stamp: "Formatted at 2026-09-26 16:44:57 on dist-test-slave-wt2j"
server_key: "8c367e4ce4aa1aac0302bf56e7fc8933"
server_key_iv: "b63e75c70ddd2dfca5a3da162caec950"
server_key_version: "encryptionkey@0"
I20260926 16:44:57.933449 20613 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260926 16:44:57.938910 20628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:57.941223 20613 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260926 16:44:57.941514 20613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "e35b57ae095742938433b3962e07d1b9"
format_stamp: "Formatted at 2026-09-26 16:44:57 on dist-test-slave-wt2j"
server_key: "8c367e4ce4aa1aac0302bf56e7fc8933"
server_key_iv: "b63e75c70ddd2dfca5a3da162caec950"
server_key_version: "encryptionkey@0"
I20260926 16:44:57.941807 20613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:57.975270 20613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:57.980937 20613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:57.981338 20613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:57.982596 20613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:57.984360 20613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:57.984479 20613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:57.984665 20613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:57.984759 20613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:58.030598 20613 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:37241
I20260926 16:44:58.030601 20740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:37241 every 8 connection(s)
I20260926 16:44:58.032290 20613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:44:58.042616 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 20613
I20260926 16:44:58.042855 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance
I20260926 16:44:58.043483 10171 external_mini_cluster.cc:1428] Setting key a61c5466ce8030862928957ccdd6a319
Sep 26 16:44:58 dist-test-slave-wt2j krb5kdc[20507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441097, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:44:58.055693 20743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:58.035461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:36113 (local address 127.9.238.254:44201)
0926 16:44:58.036176 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:58.036182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:58.037428 (+ 1246us) server_negotiation.cc:407] Connection header received
0926 16:44:58.039468 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:58.039474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:58.039615 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:58.039733 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:58.041450 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:58.042253 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:58.044630 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:58.045149 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:58.045264 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:58.049673 (+ 4409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:58.049678 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:58.049682 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:58.049717 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:58.052422 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:58.053234 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:58.053244 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:58.053247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:58.053363 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:58.053799 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:58.053804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:58.053807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:58.054098 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:58.054272 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:58.055028 (+ 756us) server_negotiation.cc:299] Negotiation successful
0926 16:44:58.055255 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":295,"threads_started":1}
I20260926 16:44:58.056845 20744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:58.036253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:44201 (local address 127.9.238.193:36113)
0926 16:44:58.037212 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:58.037261 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:58.039143 (+ 1882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:58.040020 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:58.040028 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:58.040615 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:58.041265 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:58.041282 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:58.042505 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:58.042510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:58.044405 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:58.044432 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:58.045446 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:58.045455 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:58.045812 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:58.046816 (+ 1004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:58.046853 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:58.049412 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:58.052615 (+ 3203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:58.052625 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:58.052645 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:58.053099 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:58.053471 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:58.053477 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:58.053480 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:58.053679 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:58.054227 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:58.054236 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:58.054648 (+ 412us) client_negotiation.cc:769] Sending connection context
0926 16:44:58.054940 (+ 292us) client_negotiation.cc:240] Negotiation successful
0926 16:44:58.055530 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":341,"threads_started":1}
I20260926 16:44:58.059079 20741 heartbeater.cc:347] Connected to a master server at 127.9.238.254:44201
I20260926 16:44:58.059692 20741 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:58.060956 20741 heartbeater.cc:511] Master 127.9.238.254:44201 requested a full tablet report, sending...
I20260926 16:44:58.064105 20552 ts_manager.cc:194] Registered new tserver with Master: e35b57ae095742938433b3962e07d1b9 (127.9.238.193:37241)
I20260926 16:44:58.065714 20552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.193@KRBTEST.COM'} at 127.9.238.193:36113
WARNING: no policy specified for kudu/127.9.238.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
I20260926 16:44:58.116505 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:0
--local_ip_for_outbound_sockets=127.9.238.194
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44201
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:44:58.514987 20747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:58.515516 20747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:58.515723 20747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:58.523473 20747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:58.523667 20747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:58.523789 20747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:44:58.533872 20747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:58.535348 20747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:58.536864 20747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:58.552685 20755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:58.552775 20752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:58.552804 20753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:58.554196 20747 server_base.cc:1034] running on GCE node
I20260926 16:44:58.554956 20747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:58.556262 20747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:58.557603 20747 hybrid_clock.cc:648] HybridClock initialized: now 1790441098557511 us; error 99 us; skew 500 ppm
I20260926 16:44:58.557982 20747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:58 dist-test-slave-wt2j krb5kdc[20507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441098, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:58.561746 20747 init.cc:377] Logged in from keytab as kudu/127.9.238.194@KRBTEST.COM (short username kudu)
I20260926 16:44:58.563819 20747 webserver.cc:460] Webserver started at http://127.9.238.194:43067/ using document root <none> and password file <none>
I20260926 16:44:58.564707 20747 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:58.564863 20747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:58.565210 20747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:58.568476 20747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/data/instance:
uuid: "89230c15356c45f7bd9b19f02ab6b91a"
format_stamp: "Formatted at 2026-09-26 16:44:58 on dist-test-slave-wt2j"
server_key: "b6125d6ca82d591b55b209822ea1fd29"
server_key_iv: "baee19b4a23c2546d028ab3ad3bb2491"
server_key_version: "encryptionkey@0"
I20260926 16:44:58.569312 20747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance:
uuid: "89230c15356c45f7bd9b19f02ab6b91a"
format_stamp: "Formatted at 2026-09-26 16:44:58 on dist-test-slave-wt2j"
server_key: "b6125d6ca82d591b55b209822ea1fd29"
server_key_iv: "baee19b4a23c2546d028ab3ad3bb2491"
server_key_version: "encryptionkey@0"
I20260926 16:44:58.575759 20747 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 16:44:58.580367 20762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:58.582134 20747 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:44:58.582386 20747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "89230c15356c45f7bd9b19f02ab6b91a"
format_stamp: "Formatted at 2026-09-26 16:44:58 on dist-test-slave-wt2j"
server_key: "b6125d6ca82d591b55b209822ea1fd29"
server_key_iv: "baee19b4a23c2546d028ab3ad3bb2491"
server_key_version: "encryptionkey@0"
I20260926 16:44:58.582652 20747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:58.601869 20747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:58.606963 20747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:58.607363 20747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:58.608563 20747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:58.610675 20747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:58.610801 20747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:58.610984 20747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:58.611063 20747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:58.651273 20747 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:36883
I20260926 16:44:58.651345 20874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:36883 every 8 connection(s)
I20260926 16:44:58.653060 20747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:44:58.659121 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 20747
I20260926 16:44:58.659427 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance
I20260926 16:44:58.660107 10171 external_mini_cluster.cc:1428] Setting key 9c387746820773317f9823a8048bd703
Sep 26 16:44:58 dist-test-slave-wt2j krb5kdc[20507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441098, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:44:58.678376 20877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:58.656718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:33585 (local address 127.9.238.254:44201)
0926 16:44:58.657319 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:58.657324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:58.658740 (+ 1416us) server_negotiation.cc:407] Connection header received
0926 16:44:58.661060 (+ 2320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:58.661068 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:58.661190 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:58.661369 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:58.663502 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:58.664277 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:58.665835 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:58.666252 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:58.666387 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:58.670096 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:58.670103 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:58.670107 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:58.670145 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:58.673445 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:58.675079 (+ 1634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:58.675085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:58.675087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:58.675228 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:58.676019 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:58.676023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:58.676026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:58.676354 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:58.676489 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:58.677841 (+ 1352us) server_negotiation.cc:299] Negotiation successful
0926 16:44:58.678080 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":307,"threads_started":1}
I20260926 16:44:58.680625 20878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:58.657586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:44201 (local address 127.9.238.194:33585)
0926 16:44:58.658528 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:58.658581 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:58.660624 (+ 2043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:58.661725 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:58.661737 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:58.662620 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:58.663300 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:58.663319 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:58.664446 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:58.664451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:58.665685 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:58.665697 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:58.666488 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:58.666494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:58.666722 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:58.667626 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:58.667664 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:58.669849 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:58.673695 (+ 3846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:58.673713 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:58.673744 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:58.674810 (+ 1066us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:58.675394 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:58.675403 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:58.675412 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:58.675858 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:58.676526 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:58.676541 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:58.677283 (+ 742us) client_negotiation.cc:769] Sending connection context
0926 16:44:58.677707 (+ 424us) client_negotiation.cc:240] Negotiation successful
0926 16:44:58.678548 (+ 841us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":399,"threads_started":1}
I20260926 16:44:58.682693 20875 heartbeater.cc:347] Connected to a master server at 127.9.238.254:44201
I20260926 16:44:58.683272 20875 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:58.684239 20875 heartbeater.cc:511] Master 127.9.238.254:44201 requested a full tablet report, sending...
I20260926 16:44:58.685997 20552 ts_manager.cc:194] Registered new tserver with Master: 89230c15356c45f7bd9b19f02ab6b91a (127.9.238.194:36883)
I20260926 16:44:58.686965 20552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.194@KRBTEST.COM'} at 127.9.238.194:33585
WARNING: no policy specified for kudu/127.9.238.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
I20260926 16:44:58.732833 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:0
--local_ip_for_outbound_sockets=127.9.238.195
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44201
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:35825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:44:59.070502 20741 heartbeater.cc:503] Master 127.9.238.254:44201 was elected leader, sending a full tablet report...
W20260926 16:44:59.083632 20881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:44:59.084012 20881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:44:59.084234 20881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:44:59.091899 20881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:44:59.092028 20881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:44:59.092208 20881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:44:59.102326 20881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:35825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:44:59.103811 20881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:44:59.105185 20881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:44:59.120239 20886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:59.120723 20889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:44:59.121176 20887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:44:59.121766 20881 server_base.cc:1034] running on GCE node
I20260926 16:44:59.122565 20881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:44:59.123735 20881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:44:59.125005 20881 hybrid_clock.cc:648] HybridClock initialized: now 1790441099124922 us; error 51 us; skew 500 ppm
I20260926 16:44:59.125370 20881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:44:59 dist-test-slave-wt2j krb5kdc[20507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441099, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:44:59.128999 20881 init.cc:377] Logged in from keytab as kudu/127.9.238.195@KRBTEST.COM (short username kudu)
I20260926 16:44:59.130633 20881 webserver.cc:460] Webserver started at http://127.9.238.195:35515/ using document root <none> and password file <none>
I20260926 16:44:59.131340 20881 fs_manager.cc:359] Metadata directory not provided
I20260926 16:44:59.131450 20881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:44:59.131784 20881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:44:59.134718 20881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/data/instance:
uuid: "274a377ff5f44f1ca58234c8958bdc51"
format_stamp: "Formatted at 2026-09-26 16:44:59 on dist-test-slave-wt2j"
server_key: "284402d689d991a8c8aefa764b10dbd1"
server_key_iv: "b307c4a7a031b80025fbe1dc01d830e6"
server_key_version: "encryptionkey@0"
I20260926 16:44:59.135466 20881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance:
uuid: "274a377ff5f44f1ca58234c8958bdc51"
format_stamp: "Formatted at 2026-09-26 16:44:59 on dist-test-slave-wt2j"
server_key: "284402d689d991a8c8aefa764b10dbd1"
server_key_iv: "b307c4a7a031b80025fbe1dc01d830e6"
server_key_version: "encryptionkey@0"
I20260926 16:44:59.141770 20881 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260926 16:44:59.146202 20896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:59.148562 20881 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 16:44:59.148775 20881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "274a377ff5f44f1ca58234c8958bdc51"
format_stamp: "Formatted at 2026-09-26 16:44:59 on dist-test-slave-wt2j"
server_key: "284402d689d991a8c8aefa764b10dbd1"
server_key_iv: "b307c4a7a031b80025fbe1dc01d830e6"
server_key_version: "encryptionkey@0"
I20260926 16:44:59.148983 20881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:44:59.161098 20881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:44:59.165879 20881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:44:59.166265 20881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:44:59.167502 20881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:44:59.169008 20881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:44:59.169091 20881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:59.169234 20881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:44:59.169287 20881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:44:59.208398 20881 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:45329
I20260926 16:44:59.208608 21008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:45329 every 8 connection(s)
I20260926 16:44:59.209820 20881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:44:59.218866 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 20881
I20260926 16:44:59.219148 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance
I20260926 16:44:59.219681 10171 external_mini_cluster.cc:1428] Setting key 026e28fca3f3bb82e284d05c613af1fb
Sep 26 16:44:59 dist-test-slave-wt2j krb5kdc[20507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441099, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:44:59.235667 21011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:59.213433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:34943 (local address 127.9.238.254:44201)
0926 16:44:59.214073 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:59.214080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:59.215270 (+ 1190us) server_negotiation.cc:407] Connection header received
0926 16:44:59.217472 (+ 2202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:59.217477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:59.217564 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:59.217694 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:59.219798 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.220880 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.223400 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.224060 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.224186 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:59.228635 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:59.228641 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:59.228644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:59.228688 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:59.231653 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:59.232655 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:59.232659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:59.232662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:59.232808 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:59.233476 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:59.233480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:59.233482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:59.233756 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:59.233886 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:59.234996 (+ 1110us) server_negotiation.cc:299] Negotiation successful
0926 16:44:59.235218 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":349,"threads_started":1}
I20260926 16:44:59.237318 21012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:59.213948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:44201 (local address 127.9.238.195:34943)
0926 16:44:59.215038 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:59.215113 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:59.217119 (+ 2006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:59.218003 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:59.218012 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:59.218677 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:44:59.219540 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:59.219616 (+ 76us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.221065 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.221071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:59.223212 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:59.223226 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.224311 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.224316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:59.224587 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:59.225470 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:44:59.225498 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:44:59.228317 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:44:59.231875 (+ 3558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:59.231885 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:59.231906 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:59.232463 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:59.232990 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:44:59.232994 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:44:59.232997 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:44:59.233242 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:44:59.233905 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:44:59.233912 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:44:59.234506 (+ 594us) client_negotiation.cc:769] Sending connection context
0926 16:44:59.234854 (+ 348us) client_negotiation.cc:240] Negotiation successful
0926 16:44:59.235633 (+ 779us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":369,"threads_started":1}
I20260926 16:44:59.238905 21009 heartbeater.cc:347] Connected to a master server at 127.9.238.254:44201
I20260926 16:44:59.239447 21009 heartbeater.cc:464] Registering TS with master...
I20260926 16:44:59.240337 21009 heartbeater.cc:511] Master 127.9.238.254:44201 requested a full tablet report, sending...
I20260926 16:44:59.242312 20552 ts_manager.cc:194] Registered new tserver with Master: 274a377ff5f44f1ca58234c8958bdc51 (127.9.238.195:45329)
I20260926 16:44:59.243317 20552 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.195@KRBTEST.COM'} at 127.9.238.195:34943
I20260926 16:44:59.249883 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:44:59.269423 21011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:59.255276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51012 (local address 127.9.238.254:44201)
0926 16:44:59.255496 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:59.255502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:59.256219 (+ 717us) server_negotiation.cc:407] Connection header received
0926 16:44:59.256389 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:59.256393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:59.256472 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:59.256587 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:59.258250 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.258990 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.260398 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.260809 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.260913 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:59.263438 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:59.263443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:59.263447 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:59.263480 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:59.266438 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:59.267391 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:59.267395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:59.267397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:59.267526 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:59.268024 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:59.268027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:59.268029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:59.268378 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:59.268558 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:59.268990 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 16:44:59.269168 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 16:44:59.274255 20552 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51012:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:44:59.281342 20552 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:44:59.311761 21022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:59.296367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:37241 (local address 127.0.0.1:36378)
0926 16:44:59.297619 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:59.297660 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:59.297865 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:59.300491 (+ 2626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:59.300505 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:59.300549 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:59.301354 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:59.301377 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.304070 (+ 2693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.304080 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:59.307410 (+ 3330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:59.307434 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.310277 (+ 2843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.310288 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:59.310574 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:59.310663 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 16:44:59.310855 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 16:44:59.311156 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"mutex_wait_us":35,"thread_start_us":789,"threads_started":1}
I20260926 16:44:59.312712 21023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:59.296777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36378 (local address 127.9.238.193:37241)
0926 16:44:59.299532 (+ 2755us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:59.299568 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:59.299609 (+ 41us) server_negotiation.cc:407] Connection header received
0926 16:44:59.299763 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:59.299777 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:59.300170 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:59.301003 (+ 833us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:59.301785 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.303841 (+ 2056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.307716 (+ 3875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.309649 (+ 1933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.309805 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:59.310101 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:59.312044 (+ 1943us) server_negotiation.cc:299] Negotiation successful
0926 16:44:59.312318 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2492,"thread_start_us":2282,"threads_started":1}
I20260926 16:44:59.313210 21024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:59.296922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:45329 (local address 127.0.0.1:49226)
0926 16:44:59.298509 (+ 1587us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:59.298532 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:59.298707 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:59.300771 (+ 2064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:59.300787 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:59.300836 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:59.302053 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:59.302072 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.305133 (+ 3061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.305143 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:59.308732 (+ 3589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:59.308757 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.312342 (+ 3585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.312349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:59.312528 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:59.312562 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 16:44:59.312655 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 16:44:59.312814 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1427,"mutex_wait_us":40,"thread_start_us":2654,"threads_started":1}
I20260926 16:44:59.314235 21025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:59.297790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49226 (local address 127.9.238.195:45329)
0926 16:44:59.299835 (+ 2045us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:59.299845 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:59.299895 (+ 50us) server_negotiation.cc:407] Connection header received
0926 16:44:59.300039 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:59.300052 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:59.300416 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:59.302317 (+ 1901us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:59.302783 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.304891 (+ 2108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.309021 (+ 4130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.311253 (+ 2232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.312267 (+ 1014us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:59.312559 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:59.313617 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0926 16:44:59.313833 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1867,"thread_start_us":1687,"threads_started":1}
I20260926 16:44:59.317881 20944 tablet_service.cc:1467] Processing CreateTablet for tablet e64a58349d8647a7a6e19c5974803068 (DEFAULT_TABLE table=test-table [id=6fe3fa24c78d4039a25f1c5eef2f077e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:59.318148 20676 tablet_service.cc:1467] Processing CreateTablet for tablet e64a58349d8647a7a6e19c5974803068 (DEFAULT_TABLE table=test-table [id=6fe3fa24c78d4039a25f1c5eef2f077e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:59.319491 20944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e64a58349d8647a7a6e19c5974803068. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:59.319980 20676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e64a58349d8647a7a6e19c5974803068. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:59.321357 21026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:59.298117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:36883 (local address 127.0.0.1:34798)
0926 16:44:59.311869 (+ 13752us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:59.311894 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:59.312055 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:59.313047 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:59.313652 (+ 605us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:59.313685 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:59.314095 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:59.314109 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.315960 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.315968 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:59.318046 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:59.318062 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.320231 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.320238 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:59.320427 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:59.320463 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 16:44:59.320583 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 16:44:59.320785 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13584,"thread_start_us":13350,"threads_started":1}
I20260926 16:44:59.321424 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:59.298370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34798 (local address 127.9.238.194:36883)
0926 16:44:59.311743 (+ 13373us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:59.311755 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:59.311989 (+ 234us) server_negotiation.cc:407] Connection header received
0926 16:44:59.312284 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:59.312294 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:59.312598 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:59.312817 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:59.314281 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.315786 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.318242 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.319702 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.319842 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:59.320054 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:59.320754 (+ 700us) server_negotiation.cc:299] Negotiation successful
0926 16:44:59.321035 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13139,"thread_start_us":13371,"threads_started":1}
I20260926 16:44:59.326412 20810 tablet_service.cc:1467] Processing CreateTablet for tablet e64a58349d8647a7a6e19c5974803068 (DEFAULT_TABLE table=test-table [id=6fe3fa24c78d4039a25f1c5eef2f077e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:44:59.328240 20810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e64a58349d8647a7a6e19c5974803068. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:44:59.334146 21028 tablet_bootstrap.cc:492] T e64a58349d8647a7a6e19c5974803068 P 274a377ff5f44f1ca58234c8958bdc51: Bootstrap starting.
I20260926 16:44:59.336272 21029 tablet_bootstrap.cc:492] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9: Bootstrap starting.
I20260926 16:44:59.337930 21028 tablet_bootstrap.cc:654] T e64a58349d8647a7a6e19c5974803068 P 274a377ff5f44f1ca58234c8958bdc51: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:59.339342 21028 log.cc:824] T e64a58349d8647a7a6e19c5974803068 P 274a377ff5f44f1ca58234c8958bdc51: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:59.340162 21029 tablet_bootstrap.cc:654] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:59.341277 21029 log.cc:824] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:59.343747 21028 tablet_bootstrap.cc:492] T e64a58349d8647a7a6e19c5974803068 P 274a377ff5f44f1ca58234c8958bdc51: No bootstrap required, opened a new log
I20260926 16:44:59.344152 21028 ts_tablet_manager.cc:1397] T e64a58349d8647a7a6e19c5974803068 P 274a377ff5f44f1ca58234c8958bdc51: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260926 16:44:59.347335 21032 tablet_bootstrap.cc:492] T e64a58349d8647a7a6e19c5974803068 P 89230c15356c45f7bd9b19f02ab6b91a: Bootstrap starting.
I20260926 16:44:59.349609 21029 tablet_bootstrap.cc:492] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9: No bootstrap required, opened a new log
I20260926 16:44:59.349989 21029 ts_tablet_manager.cc:1397] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20260926 16:44:59.351221 21032 tablet_bootstrap.cc:654] T e64a58349d8647a7a6e19c5974803068 P 89230c15356c45f7bd9b19f02ab6b91a: Neither blocks nor log segments found. Creating new log.
I20260926 16:44:59.352632 21032 log.cc:824] T e64a58349d8647a7a6e19c5974803068 P 89230c15356c45f7bd9b19f02ab6b91a: Log is configured to *not* fsync() on all Append() calls
I20260926 16:44:59.357791 21028 raft_consensus.cc:348] T e64a58349d8647a7a6e19c5974803068 P 274a377ff5f44f1ca58234c8958bdc51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35b57ae095742938433b3962e07d1b9" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 37241 } } peers { permanent_uuid: "274a377ff5f44f1ca58234c8958bdc51" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45329 } } peers { permanent_uuid: "89230c15356c45f7bd9b19f02ab6b91a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36883 } }
I20260926 16:44:59.358451 21028 raft_consensus.cc:374] T e64a58349d8647a7a6e19c5974803068 P 274a377ff5f44f1ca58234c8958bdc51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:59.358582 21028 raft_consensus.cc:729] T e64a58349d8647a7a6e19c5974803068 P 274a377ff5f44f1ca58234c8958bdc51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274a377ff5f44f1ca58234c8958bdc51, State: Initialized, Role: FOLLOWER
I20260926 16:44:59.359368 21028 consensus_queue.cc:260] T e64a58349d8647a7a6e19c5974803068 P 274a377ff5f44f1ca58234c8958bdc51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35b57ae095742938433b3962e07d1b9" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 37241 } } peers { permanent_uuid: "274a377ff5f44f1ca58234c8958bdc51" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45329 } } peers { permanent_uuid: "89230c15356c45f7bd9b19f02ab6b91a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36883 } }
I20260926 16:44:59.359292 21029 raft_consensus.cc:348] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35b57ae095742938433b3962e07d1b9" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 37241 } } peers { permanent_uuid: "274a377ff5f44f1ca58234c8958bdc51" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45329 } } peers { permanent_uuid: "89230c15356c45f7bd9b19f02ab6b91a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36883 } }
I20260926 16:44:59.359864 21029 raft_consensus.cc:374] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:59.359990 21029 raft_consensus.cc:729] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e35b57ae095742938433b3962e07d1b9, State: Initialized, Role: FOLLOWER
I20260926 16:44:59.360759 21009 heartbeater.cc:503] Master 127.9.238.254:44201 was elected leader, sending a full tablet report...
I20260926 16:44:59.360772 21029 consensus_queue.cc:260] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35b57ae095742938433b3962e07d1b9" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 37241 } } peers { permanent_uuid: "274a377ff5f44f1ca58234c8958bdc51" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45329 } } peers { permanent_uuid: "89230c15356c45f7bd9b19f02ab6b91a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36883 } }
I20260926 16:44:59.361433 21028 ts_tablet_manager.cc:1428] T e64a58349d8647a7a6e19c5974803068 P 274a377ff5f44f1ca58234c8958bdc51: Time spent starting tablet: real 0.017s user 0.011s sys 0.005s
I20260926 16:44:59.363180 21029 ts_tablet_manager.cc:1428] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260926 16:44:59.363602 21032 tablet_bootstrap.cc:492] T e64a58349d8647a7a6e19c5974803068 P 89230c15356c45f7bd9b19f02ab6b91a: No bootstrap required, opened a new log
I20260926 16:44:59.363965 21032 ts_tablet_manager.cc:1397] T e64a58349d8647a7a6e19c5974803068 P 89230c15356c45f7bd9b19f02ab6b91a: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
I20260926 16:44:59.373440 21032 raft_consensus.cc:348] T e64a58349d8647a7a6e19c5974803068 P 89230c15356c45f7bd9b19f02ab6b91a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35b57ae095742938433b3962e07d1b9" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 37241 } } peers { permanent_uuid: "274a377ff5f44f1ca58234c8958bdc51" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45329 } } peers { permanent_uuid: "89230c15356c45f7bd9b19f02ab6b91a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36883 } }
I20260926 16:44:59.373968 21032 raft_consensus.cc:374] T e64a58349d8647a7a6e19c5974803068 P 89230c15356c45f7bd9b19f02ab6b91a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:44:59.374079 21032 raft_consensus.cc:729] T e64a58349d8647a7a6e19c5974803068 P 89230c15356c45f7bd9b19f02ab6b91a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89230c15356c45f7bd9b19f02ab6b91a, State: Initialized, Role: FOLLOWER
I20260926 16:44:59.374845 21032 consensus_queue.cc:260] T e64a58349d8647a7a6e19c5974803068 P 89230c15356c45f7bd9b19f02ab6b91a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35b57ae095742938433b3962e07d1b9" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 37241 } } peers { permanent_uuid: "274a377ff5f44f1ca58234c8958bdc51" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45329 } } peers { permanent_uuid: "89230c15356c45f7bd9b19f02ab6b91a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36883 } }
I20260926 16:44:59.376219 20875 heartbeater.cc:503] Master 127.9.238.254:44201 was elected leader, sending a full tablet report...
I20260926 16:44:59.376813 21032 ts_tablet_manager.cc:1428] T e64a58349d8647a7a6e19c5974803068 P 89230c15356c45f7bd9b19f02ab6b91a: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
W20260926 16:44:59.405392 20876 tablet.cc:2369] T e64a58349d8647a7a6e19c5974803068 P 89230c15356c45f7bd9b19f02ab6b91a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:44:59.411180 21035 raft_consensus.cc:482] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:44:59.411777 21035 raft_consensus.cc:504] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35b57ae095742938433b3962e07d1b9" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 37241 } } peers { permanent_uuid: "274a377ff5f44f1ca58234c8958bdc51" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45329 } } peers { permanent_uuid: "89230c15356c45f7bd9b19f02ab6b91a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36883 } }
I20260926 16:44:59.414033 21035 leader_election.cc:288] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 274a377ff5f44f1ca58234c8958bdc51 (127.9.238.195:45329), 89230c15356c45f7bd9b19f02ab6b91a (127.9.238.194:36883)
I20260926 16:44:59.422572 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:59.414741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:50089 (local address 127.9.238.194:36883)
0926 16:44:59.415102 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:59.415111 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:59.416187 (+ 1076us) server_negotiation.cc:407] Connection header received
0926 16:44:59.416510 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:59.416516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:59.416582 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:59.416700 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:59.417430 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.418246 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.420464 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.421456 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.421645 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:59.421727 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:59.422112 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 16:44:59.422243 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260926 16:44:59.422716 21038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:59.414791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:36883 (local address 127.9.238.193:50089)
0926 16:44:59.416061 (+ 1270us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:59.416080 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:59.416313 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:59.416723 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:59.416728 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:59.416760 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:59.417259 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:59.417273 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.418388 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.418396 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:59.420292 (+ 1896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:59.420306 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.421669 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.421677 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:59.421916 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:59.421945 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 16:44:59.422069 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 16:44:59.422229 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1097,"thread_start_us":759,"threads_started":1}
I20260926 16:44:59.423290 21025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:59.414822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:36389 (local address 127.9.238.195:45329)
0926 16:44:59.415089 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:59.415098 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:59.416342 (+ 1244us) server_negotiation.cc:407] Connection header received
0926 16:44:59.416581 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:59.416586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:59.416661 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:59.416786 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:44:59.417527 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.418418 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.420760 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.421969 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.422177 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:59.422298 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:59.422840 (+ 542us) server_negotiation.cc:299] Negotiation successful
0926 16:44:59.422993 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 16:44:59.424022 21037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:59.414829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:45329 (local address 127.9.238.193:36389)
0926 16:44:59.416177 (+ 1348us) negotiation.cc:107] Waiting for socket to connect
0926 16:44:59.416197 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:44:59.416374 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:44:59.416803 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:44:59.416808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:44:59.416839 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:44:59.417331 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:59.417341 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.418599 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.418604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:59.420482 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:44:59.420495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.422337 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.422346 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:44:59.422575 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:59.422604 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 16:44:59.423427 (+ 823us) client_negotiation.cc:240] Negotiation successful
0926 16:44:59.423644 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1211,"thread_start_us":609,"threads_started":1}
I20260926 16:44:59.423971 20830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e64a58349d8647a7a6e19c5974803068" candidate_uuid: "e35b57ae095742938433b3962e07d1b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89230c15356c45f7bd9b19f02ab6b91a" is_pre_election: true
I20260926 16:44:59.424609 20830 raft_consensus.cc:2454] T e64a58349d8647a7a6e19c5974803068 P 89230c15356c45f7bd9b19f02ab6b91a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e35b57ae095742938433b3962e07d1b9 in term 0.
I20260926 16:44:59.425381 20631 leader_election.cc:302] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89230c15356c45f7bd9b19f02ab6b91a, e35b57ae095742938433b3962e07d1b9; no voters:
I20260926 16:44:59.425299 20964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e64a58349d8647a7a6e19c5974803068" candidate_uuid: "e35b57ae095742938433b3962e07d1b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274a377ff5f44f1ca58234c8958bdc51" is_pre_election: true
I20260926 16:44:59.425829 21035 raft_consensus.cc:2781] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:44:59.425997 21035 raft_consensus.cc:482] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:44:59.425993 20964 raft_consensus.cc:2454] T e64a58349d8647a7a6e19c5974803068 P 274a377ff5f44f1ca58234c8958bdc51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e35b57ae095742938433b3962e07d1b9 in term 0.
I20260926 16:44:59.426126 21035 raft_consensus.cc:3037] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:59.428963 21035 raft_consensus.cc:504] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35b57ae095742938433b3962e07d1b9" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 37241 } } peers { permanent_uuid: "274a377ff5f44f1ca58234c8958bdc51" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45329 } } peers { permanent_uuid: "89230c15356c45f7bd9b19f02ab6b91a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36883 } }
I20260926 16:44:59.429849 21035 leader_election.cc:288] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9 [CANDIDATE]: Term 1 election: Requested vote from peers 274a377ff5f44f1ca58234c8958bdc51 (127.9.238.195:45329), 89230c15356c45f7bd9b19f02ab6b91a (127.9.238.194:36883)
I20260926 16:44:59.430548 20830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e64a58349d8647a7a6e19c5974803068" candidate_uuid: "e35b57ae095742938433b3962e07d1b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89230c15356c45f7bd9b19f02ab6b91a"
I20260926 16:44:59.430544 20964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e64a58349d8647a7a6e19c5974803068" candidate_uuid: "e35b57ae095742938433b3962e07d1b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274a377ff5f44f1ca58234c8958bdc51"
I20260926 16:44:59.430799 20964 raft_consensus.cc:3037] T e64a58349d8647a7a6e19c5974803068 P 274a377ff5f44f1ca58234c8958bdc51 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:59.430799 20830 raft_consensus.cc:3037] T e64a58349d8647a7a6e19c5974803068 P 89230c15356c45f7bd9b19f02ab6b91a [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:44:59.434479 20830 raft_consensus.cc:2454] T e64a58349d8647a7a6e19c5974803068 P 89230c15356c45f7bd9b19f02ab6b91a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e35b57ae095742938433b3962e07d1b9 in term 1.
I20260926 16:44:59.434470 20964 raft_consensus.cc:2454] T e64a58349d8647a7a6e19c5974803068 P 274a377ff5f44f1ca58234c8958bdc51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e35b57ae095742938433b3962e07d1b9 in term 1.
I20260926 16:44:59.435102 20630 leader_election.cc:302] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 274a377ff5f44f1ca58234c8958bdc51, e35b57ae095742938433b3962e07d1b9; no voters:
I20260926 16:44:59.435614 21035 raft_consensus.cc:2781] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:44:59.436350 21035 raft_consensus.cc:686] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9 [term 1 LEADER]: Becoming Leader. State: Replica: e35b57ae095742938433b3962e07d1b9, State: Running, Role: LEADER
I20260926 16:44:59.437314 21035 consensus_queue.cc:237] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35b57ae095742938433b3962e07d1b9" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 37241 } } peers { permanent_uuid: "274a377ff5f44f1ca58234c8958bdc51" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45329 } } peers { permanent_uuid: "89230c15356c45f7bd9b19f02ab6b91a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36883 } }
I20260926 16:44:59.443019 20552 catalog_manager.cc:5495] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e35b57ae095742938433b3962e07d1b9 (127.9.238.193). New cstate: current_term: 1 leader_uuid: "e35b57ae095742938433b3962e07d1b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e35b57ae095742938433b3962e07d1b9" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 37241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "274a377ff5f44f1ca58234c8958bdc51" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89230c15356c45f7bd9b19f02ab6b91a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36883 } health_report { overall_health: UNKNOWN } } }
W20260926 16:44:59.461879 21010 tablet.cc:2369] T e64a58349d8647a7a6e19c5974803068 P 274a377ff5f44f1ca58234c8958bdc51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:44:59.475289 21023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:59.468974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36382 (local address 127.9.238.193:37241)
0926 16:44:59.469193 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:59.469199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:59.469221 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:44:59.469455 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:59.469460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:59.469573 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:59.469786 (+ 213us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:44:59.470177 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.471043 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.472663 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.473162 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.473308 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:59.473956 (+ 648us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:44:59.474625 (+ 669us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:44:59.474775 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:59.474922 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 16:44:59.475012 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 16:44:59.486562 20830 raft_consensus.cc:1261] T e64a58349d8647a7a6e19c5974803068 P 89230c15356c45f7bd9b19f02ab6b91a [term 1 FOLLOWER]: Refusing update from remote peer e35b57ae095742938433b3962e07d1b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:44:59.486543 20964 raft_consensus.cc:1261] T e64a58349d8647a7a6e19c5974803068 P 274a377ff5f44f1ca58234c8958bdc51 [term 1 FOLLOWER]: Refusing update from remote peer e35b57ae095742938433b3962e07d1b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:44:59.488170 21039 consensus_queue.cc:1035] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "274a377ff5f44f1ca58234c8958bdc51" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 45329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:44:59.488674 21035 consensus_queue.cc:1035] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89230c15356c45f7bd9b19f02ab6b91a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 36883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:44:59.503600 21042 mvcc.cc:205] Tried to move back new op lower bound from 7333646743481307136 to 7333646743299596288. Current Snapshot: MvccSnapshot[applied={T|T < 7333646743481307136}]
I20260926 16:44:59.523363 20550 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:44:59.523836 20550 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:44:59.529214 20550 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bad906ec6df4e37b3e86eb02a1dc5bb: Sending DeleteTablet for 3 replicas of tablet e64a58349d8647a7a6e19c5974803068
I20260926 16:44:59.530963 20676 tablet_service.cc:1514] Processing DeleteTablet for tablet e64a58349d8647a7a6e19c5974803068 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:59 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:36378
I20260926 16:44:59.530941 20944 tablet_service.cc:1514] Processing DeleteTablet for tablet e64a58349d8647a7a6e19c5974803068 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:59 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:49226
I20260926 16:44:59.530951 20810 tablet_service.cc:1514] Processing DeleteTablet for tablet e64a58349d8647a7a6e19c5974803068 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:44:59 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:34798
I20260926 16:44:59.534191 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 20613
I20260926 16:44:59.548312 21054 tablet_replica.cc:330] T e64a58349d8647a7a6e19c5974803068 P 89230c15356c45f7bd9b19f02ab6b91a: stopping tablet replica
I20260926 16:44:59.548889 21054 raft_consensus.cc:2229] T e64a58349d8647a7a6e19c5974803068 P 89230c15356c45f7bd9b19f02ab6b91a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:44:59.549517 21054 raft_consensus.cc:2258] T e64a58349d8647a7a6e19c5974803068 P 89230c15356c45f7bd9b19f02ab6b91a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:44:59.551964 21054 ts_tablet_manager.cc:1905] T e64a58349d8647a7a6e19c5974803068 P 89230c15356c45f7bd9b19f02ab6b91a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:44:59.560093 21054 ts_tablet_manager.cc:1918] T e64a58349d8647a7a6e19c5974803068 P 89230c15356c45f7bd9b19f02ab6b91a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:44:59.560300 21054 log.cc:1196] T e64a58349d8647a7a6e19c5974803068 P 89230c15356c45f7bd9b19f02ab6b91a: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-1/wal/wals/e64a58349d8647a7a6e19c5974803068
I20260926 16:44:59.560935 21054 ts_tablet_manager.cc:1939] T e64a58349d8647a7a6e19c5974803068 P 89230c15356c45f7bd9b19f02ab6b91a: Deleting consensus metadata
I20260926 16:44:59.561846 21053 tablet_replica.cc:330] T e64a58349d8647a7a6e19c5974803068 P 274a377ff5f44f1ca58234c8958bdc51: stopping tablet replica
I20260926 16:44:59.563305 20538 catalog_manager.cc:4841] TS 89230c15356c45f7bd9b19f02ab6b91a (127.9.238.194:36883): tablet e64a58349d8647a7a6e19c5974803068 (table test-table [id=6fe3fa24c78d4039a25f1c5eef2f077e]) successfully deleted
I20260926 16:44:59.563827 21052 tablet_replica.cc:330] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9: stopping tablet replica
I20260926 16:44:59.564237 21053 raft_consensus.cc:2229] T e64a58349d8647a7a6e19c5974803068 P 274a377ff5f44f1ca58234c8958bdc51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:44:59.564366 21052 raft_consensus.cc:2229] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:44:59.565075 21053 raft_consensus.cc:2258] T e64a58349d8647a7a6e19c5974803068 P 274a377ff5f44f1ca58234c8958bdc51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:44:59.565250 21052 raft_consensus.cc:2258] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:44:59.568034 21053 ts_tablet_manager.cc:1905] T e64a58349d8647a7a6e19c5974803068 P 274a377ff5f44f1ca58234c8958bdc51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:44:59.568383 21052 ts_tablet_manager.cc:1905] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 26 16:44:59 dist-test-slave-wt2j krb5kdc[20507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.193@KRBTEST.COM
I20260926 16:44:59.576828 21053 ts_tablet_manager.cc:1918] T e64a58349d8647a7a6e19c5974803068 P 274a377ff5f44f1ca58234c8958bdc51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:44:59.577256 21053 log.cc:1196] T e64a58349d8647a7a6e19c5974803068 P 274a377ff5f44f1ca58234c8958bdc51: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-2/wal/wals/e64a58349d8647a7a6e19c5974803068
I20260926 16:44:59.577435 21052 ts_tablet_manager.cc:1918] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:44:59.577699 21052 log.cc:1196] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790440988577176-10171-0/minicluster-data/ts-0/wal/wals/e64a58349d8647a7a6e19c5974803068
I20260926 16:44:59.577941 21053 ts_tablet_manager.cc:1939] T e64a58349d8647a7a6e19c5974803068 P 274a377ff5f44f1ca58234c8958bdc51: Deleting consensus metadata
I20260926 16:44:59.578619 21052 ts_tablet_manager.cc:1939] T e64a58349d8647a7a6e19c5974803068 P e35b57ae095742938433b3962e07d1b9: Deleting consensus metadata
I20260926 16:44:59.580348 20537 catalog_manager.cc:4841] TS 274a377ff5f44f1ca58234c8958bdc51 (127.9.238.195:45329): tablet e64a58349d8647a7a6e19c5974803068 (table test-table [id=6fe3fa24c78d4039a25f1c5eef2f077e]) successfully deleted
I20260926 16:44:59.580967 20538 catalog_manager.cc:4841] TS e35b57ae095742938433b3962e07d1b9 (127.9.238.193:37241): tablet e64a58349d8647a7a6e19c5974803068 (table test-table [id=6fe3fa24c78d4039a25f1c5eef2f077e]) successfully deleted
I20260926 16:44:59.585130 21023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:59.536291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36388 (local address 127.9.238.193:37241)
0926 16:44:59.536572 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:59.536582 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:59.559287 (+ 22705us) server_negotiation.cc:407] Connection header received
0926 16:44:59.560070 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:59.560081 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:59.560213 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:59.560292 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:59.564735 (+ 4443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.566152 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.568189 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.568894 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.569010 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:59.575824 (+ 6814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:59.575848 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:59.575874 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:59.575959 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:59.581175 (+ 5216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:59.582429 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:59.582437 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:59.582440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:59.582671 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:59.583223 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:59.583228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:59.583230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:59.583725 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:59.584001 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:59.584515 (+ 514us) server_negotiation.cc:299] Negotiation successful
0926 16:44:59.584728 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 16:44:59.585954 20736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:59.712152 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 20613
I20260926 16:44:59.731881 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 20747
Sep 26 16:44:59 dist-test-slave-wt2j krb5kdc[20507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.194@KRBTEST.COM
I20260926 16:44:59.750385 21027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:59.733139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34808 (local address 127.9.238.194:36883)
0926 16:44:59.733343 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:59.733348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:59.733370 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:44:59.733557 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:59.733562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:59.733636 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:59.733751 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:59.735249 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.736157 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.737645 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.738099 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.738267 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:59.743206 (+ 4939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:59.743235 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:59.743259 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:59.743318 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:59.747109 (+ 3791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:59.748026 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:59.748036 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:59.748038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:59.748201 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:59.748695 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:59.748700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:59.748702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:59.749175 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:59.749525 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:59.749854 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 16:44:59.750058 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 16:44:59.751336 20870 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:59.854580 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 20747
I20260926 16:44:59.875860 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 20881
Sep 26 16:44:59 dist-test-slave-wt2j krb5kdc[20507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.195@KRBTEST.COM
I20260926 16:44:59.893267 21025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:44:59.876910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49232 (local address 127.9.238.195:45329)
0926 16:44:59.877364 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0926 16:44:59.877370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:44:59.877398 (+ 28us) server_negotiation.cc:407] Connection header received
0926 16:44:59.877480 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:44:59.877485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:44:59.877588 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:44:59.877715 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:44:59.879419 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.880366 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.881723 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:44:59.882145 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:44:59.882250 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:44:59.887267 (+ 5017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:44:59.887285 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:44:59.887307 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:44:59.887359 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:44:59.890220 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:59.891086 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:59.891094 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:59.891096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:59.891256 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:44:59.891756 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:44:59.891761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:44:59.891763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:44:59.892141 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:44:59.892379 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 16:44:59.892800 (+ 421us) server_negotiation.cc:299] Negotiation successful
0926 16:44:59.892982 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260926 16:44:59.894178 21004 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:44:59.986657 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 20881
I20260926 16:45:00.009871 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 20520
I20260926 16:45:00.011695 20582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:45:00.122534 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 20520
2026-09-26T16:45:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4799 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:45:00 dist-test-slave-wt2j krb5kdc[21062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:45:00 dist-test-slave-wt2j krb5kdc[21062](info): set up 2 sockets
Sep 26 16:45:00 dist-test-slave-wt2j krb5kdc[21062](info): commencing operation
krb5kdc: starting...
W20260926 16:45:01.266710 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:45:01 dist-test-slave-wt2j krb5kdc[21062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:45:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:45:01Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.238.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.238.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.238.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/rangeradmin_127.9.238.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.238.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/rangeradmin_127.9.238.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.238.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.238.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.238.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/rangerlookup_127.9.238.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.238.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/rangerlookup_127.9.238.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.238.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.238.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.238.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/HTTP_127.9.238.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.238.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/HTTP_127.9.238.212@KRBTEST.COM.keytab.
I20260926 16:45:01.697249 10171 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskIzJvco/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/postgres -l logfile start
2026-09-26 16:45:04.726 UTC [21092] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 16:45:04.727 UTC [21092] LOG: listening on IPv4 address "127.9.238.212", port 44421
2026-09-26 16:45:04.732 UTC [21092] LOG: listening on Unix socket "/tmp/.s.PGSQL.44421"
2026-09-26 16:45:04.761 UTC [21095] LOG: database system was shut down at 2026-09-26 16:45:02 UTC
2026-09-26 16:45:04.767 UTC [21092] LOG: database system is ready to accept connections
I20260926 16:45:05.754266 10171 mini_postgres.cc:96] Postgres bound to 44421
2026-09-26 16:45:05.778 UTC [21103] FATAL: database "slave" does not exist
127.9.238.212:44421 - accepting connections
I20260926 16:45:05.779604 10171 mini_ranger.cc:161] Starting Ranger...
I20260926 16:45:05.814726 10171 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 16:45:06.481803 10171 mini_ranger.cc:87] Created ranger Postgres database
I20260926 16:45:06.481954 10171 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ranger-admin
2026-09-26 16:45:06,782 [I] DB FLAVOR :POSTGRES
2026-09-26 16:45:06,782 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 16:45:06,782 [I] Checking connection..
2026-09-26 16:45:06,783 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:44421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 16:45:07,094 [I] Checking connection passed.
2026-09-26 16:45:07,094 [I] --------- Verifying version history table ---------
2026-09-26 16:45:07,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:44421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 16:45:07,345 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 16:45:07,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:44421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 16:45:07,613 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 16:45:07,614 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 16:45:07,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:44421/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 16:45:07.894 UTC [21167] WARNING: there is no transaction in progress
2026-09-26 16:45:07,903 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 16:45:07,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:44421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 16:45:08,180 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 16:45:08,180 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 16:45:08,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:44421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 16:45:08,437 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:44421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 16:45:08,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:44421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wt2j.c.gcp-upstream.internal','N') ;"
2026-09-26 16:45:08,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:44421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 16:45:09,252 [I] Table x_portal_user does not exist in database ranger
2026-09-26 16:45:09,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:44421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 16:45:09,517 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 16:45:09,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:44421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 16:45:09,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:44421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 16:45:10,029 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 16:45:10,029 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:44421/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 16:45:11.171 UTC [21275] WARNING: there is no transaction in progress
2026-09-26 16:45:11.191 UTC [21275] WARNING: there is no transaction in progress
2026-09-26 16:45:11.209 UTC [21275] WARNING: there is no transaction in progress
2026-09-26 16:45:11.227 UTC [21275] WARNING: there is no transaction in progress
2026-09-26 16:45:11.245 UTC [21275] WARNING: there is no transaction in progress
2026-09-26 16:45:11.264 UTC [21275] WARNING: there is no transaction in progress
2026-09-26 16:45:11.325 UTC [21275] WARNING: there is no transaction in progress
2026-09-26 16:45:11.333 UTC [21275] WARNING: there is no transaction in progress
2026-09-26 16:45:11.351 UTC [21275] WARNING: there is no transaction in progress
2026-09-26 16:45:11.363 UTC [21275] WARNING: there is no transaction in progress
2026-09-26 16:45:11.377 UTC [21275] WARNING: there is no transaction in progress
2026-09-26 16:45:11.394 UTC [21275] WARNING: there is no transaction in progress
2026-09-26 16:45:11.406 UTC [21275] WARNING: there is no transaction in progress
2026-09-26 16:45:11.414 UTC [21275] WARNING: there is no transaction in progress
2026-09-26 16:45:11.423 UTC [21275] WARNING: there is no transaction in progress
2026-09-26 16:45:11.432 UTC [21275] WARNING: there is no transaction in progress
2026-09-26 16:45:12,319 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 16:45:12,319 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:44421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 16:45:12,601 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 16:45:12,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:44421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 16:45:12,881 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 16:45:12,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:44421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 16:45:13,187 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 16:45:13,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:44421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 16:45:13,488 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:44421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 16:45:13,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:44421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wt2j.c.gcp-upstream.internal';"
2026-09-26 16:45:14,037 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 16:45:14,037 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:44421/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 16:45:14,343 [I] DB_PATCHES have already been applied
I20260926 16:45:14.349905 10171 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ranger-admin:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskIzJvco/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 16:45:14.350077 10171 mini_ranger.cc:193] Using host: 127.9.238.212
I20260926 16:45:14.369298 10171 mini_ranger.cc:227] Ranger admin URL: http://127.9.238.212:47635
Sep 26, 2026 4:45:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 4:45:14 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 4:45:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 4:45:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 4:45:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 4:45:15 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 4:45:15 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 16:45:15.528301 10171 mini_ranger.cc:160] Time spent starting Ranger: real 9.749s user 0.000s sys 0.064s
Sep 26, 2026 4:45:39 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 4:45:39 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 4:45:39 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 4:45:39 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 4:45:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 4:45:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 4:45:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 4:45:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 4:45:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 4:45:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 4:45:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 4:45:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 4:45:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 4:45:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 4:45:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 4:45:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 4:45:39 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 4:45:40 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 16:45:40.970391 10171 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.238.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.238.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.238.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/rangerkms_127.9.238.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.238.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/rangerkms_127.9.238.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.238.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.238.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.238.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/HTTP_127.9.238.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.238.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/HTTP_127.9.238.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 16:45:41 dist-test-slave-wt2j krb5kdc[21062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441141, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 16:45:41.179495 10171 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 16:45:41.208333 10171 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 16:45:42.290609 10171 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 16:45:42.290907 10171 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ranger-kms
I20260926 16:45:42.291033 10171 mini_ranger_kms.cc:227] Using postgres at 127.9.238.212:44421
2026-09-26 16:45:42,758 [I] DB FLAVOR :POSTGRES
2026-09-26 16:45:42,758 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 16:45:42,758 [I] Checking connection
2026-09-26 16:45:42,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:44421/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 16:45:43,036 [I] connection success
2026-09-26 16:45:43,036 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 16:45:43,036 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 16:45:43,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:44421/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 16:45:43,307 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 16:45:43,308 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 16:45:43,308 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 16:45:43,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:44421/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 16:45:43,640 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 16:45:44.257774 10171 mini_ranger_kms.cc:310] Created kms service
I20260926 16:45:44.475021 10171 mini_ranger_kms.cc:326] Created kudu user
I20260926 16:45:44.540181 10171 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 16:45:44 dist-test-slave-wt2j krb5kdc[21062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441141, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.238.212@KRBTEST.COM
I20260926 16:45:45.014259 10171 mini_ranger_kms.cc:384] Added ranger policy
I20260926 16:45:45.014622 10171 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ranger-kms:/tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 16:45:45.014693 10171 mini_ranger_kms.cc:240] Using host: 127.9.238.212
I20260926 16:45:45.040760 10171 mini_ranger_kms.cc:276] Ranger KMS PID: 21456
I20260926 16:45:45.041002 10171 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.238.212:35827
Sep 26, 2026 4:45:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 4:45:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 4:45:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 4:45:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 4:45:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 4:45:46 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 16:45:47.463060 10171 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.284s user 0.155s sys 0.350s
I20260926 16:45:47.463316 10171 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 16:45:47.463532 10171 mini_ranger_kms.cc:401] 127.9.238.212:35827/kms/v1/keys
Sep 26 16:45:47 dist-test-slave-wt2j krb5kdc[21062](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790441147, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.238.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 16:45:48 dist-test-slave-wt2j krb5kdc[21062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441147, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.238.212@KRBTEST.COM for HTTP/127.9.238.212@KRBTEST.COM
Sep 26 16:45:48 dist-test-slave-wt2j krb5kdc[21062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441147, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.238.212@KRBTEST.COM for HTTP/127.9.238.212@KRBTEST.COM
Sep 26 16:45:51 dist-test-slave-wt2j krb5kdc[21062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:45:51.329514 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:45319
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:34771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:45319
--ranger_config_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.238.212:35827/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:45:51.744560 21506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:45:51.744887 21506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:45:51.745077 21506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:45:51.752363 21506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:45:51.752485 21506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:45:51.752568 21506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:45:51.752658 21506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:45:51.752727 21506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:45:51.763247 21506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:34771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.238.212:35827/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:45319
--ranger_config_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:45319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:45:51.764755 21506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:45:51.766161 21506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:45:51.780766 21511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:45:51.780889 21512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:45:51.780869 21514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:45:51.781221 21506 server_base.cc:1034] running on GCE node
I20260926 16:45:51.782594 21506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:45:51.784425 21506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:45:51.785701 21506 hybrid_clock.cc:648] HybridClock initialized: now 1790441151785614 us; error 85 us; skew 500 ppm
I20260926 16:45:51.786125 21506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:45:51 dist-test-slave-wt2j krb5kdc[21062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441151, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:45:51.799701 21506 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:45:51.801673 21506 webserver.cc:460] Webserver started at http://127.9.238.254:41649/ using document root <none> and password file <none>
I20260926 16:45:51.802436 21506 fs_manager.cc:359] Metadata directory not provided
I20260926 16:45:51.802591 21506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:45:51.802912 21506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:45:51 dist-test-slave-wt2j krb5kdc[21062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441151, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for HTTP/127.9.238.212@KRBTEST.COM
I20260926 16:45:51.878216 21506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "7eb85176a1e344f78f7dfd87f39206b6"
format_stamp: "Formatted at 2026-09-26 16:45:51 on dist-test-slave-wt2j"
server_key: "9ac816f36b8a542154d8e87db65eb6f7"
server_key_iv: "411fa3354f9e8fb6e7af7c4041937d1a"
server_key_version: "kuduclusterkey@0"
I20260926 16:45:51.879122 21506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "7eb85176a1e344f78f7dfd87f39206b6"
format_stamp: "Formatted at 2026-09-26 16:45:51 on dist-test-slave-wt2j"
server_key: "9ac816f36b8a542154d8e87db65eb6f7"
server_key_iv: "411fa3354f9e8fb6e7af7c4041937d1a"
server_key_version: "kuduclusterkey@0"
I20260926 16:45:51.885196 21506 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260926 16:45:51.910389 21521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:51.912245 21506 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:45:51.912459 21506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "7eb85176a1e344f78f7dfd87f39206b6"
format_stamp: "Formatted at 2026-09-26 16:45:51 on dist-test-slave-wt2j"
server_key: "9ac816f36b8a542154d8e87db65eb6f7"
server_key_iv: "411fa3354f9e8fb6e7af7c4041937d1a"
server_key_version: "kuduclusterkey@0"
I20260926 16:45:51.912739 21506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:45:51.943758 21506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:45:51.948592 21506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:45:51.948959 21506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:45:51.971792 21506 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:45319
I20260926 16:45:51.971800 21572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:45319 every 8 connection(s)
I20260926 16:45:51.973555 21506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:45:51.978737 21573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:45:51.979205 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 21506
I20260926 16:45:51.979446 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:45:51.980379 10171 external_mini_cluster.cc:1428] Setting key b0e23cd941a07e0b7ef2c2579c749cdd
I20260926 16:45:51.997697 21573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eb85176a1e344f78f7dfd87f39206b6: Bootstrap starting.
Sep 26 16:45:51 dist-test-slave-wt2j krb5kdc[21062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:45:52.004118 21573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eb85176a1e344f78f7dfd87f39206b6: Neither blocks nor log segments found. Creating new log.
I20260926 16:45:52.006234 21573 log.cc:824] T 00000000000000000000000000000000 P 7eb85176a1e344f78f7dfd87f39206b6: Log is configured to *not* fsync() on all Append() calls
I20260926 16:45:52.011576 21573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eb85176a1e344f78f7dfd87f39206b6: No bootstrap required, opened a new log
I20260926 16:45:52.013459 21576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:51.983116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32820 (local address 127.9.238.254:45319)
0926 16:45:51.984099 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:51.984112 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:51.984162 (+ 50us) server_negotiation.cc:407] Connection header received
0926 16:45:51.986889 (+ 2727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:51.986920 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:51.987461 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:51.988013 (+ 552us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:45:51.990139 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:51.991849 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:51.994081 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:51.994998 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:51.995182 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:52.001894 (+ 6712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:45:52.001913 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:45:52.001940 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:45:52.002015 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:45:52.007309 (+ 5294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:52.008701 (+ 1392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:52.008715 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:52.008728 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:52.009011 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:52.009508 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:52.009514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:52.009517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:52.010407 (+ 890us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:45:52.010781 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:52.011227 (+ 446us) server_negotiation.cc:299] Negotiation successful
0926 16:45:52.011801 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":489,"threads_started":1}
I20260926 16:45:52.021706 21573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eb85176a1e344f78f7dfd87f39206b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb85176a1e344f78f7dfd87f39206b6" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 45319 } }
I20260926 16:45:52.022205 21573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7eb85176a1e344f78f7dfd87f39206b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:45:52.022348 21573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eb85176a1e344f78f7dfd87f39206b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb85176a1e344f78f7dfd87f39206b6, State: Initialized, Role: FOLLOWER
I20260926 16:45:52.023140 21573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eb85176a1e344f78f7dfd87f39206b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb85176a1e344f78f7dfd87f39206b6" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 45319 } }
I20260926 16:45:52.023510 21573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7eb85176a1e344f78f7dfd87f39206b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:45:52.023695 21573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eb85176a1e344f78f7dfd87f39206b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:45:52.023903 21573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eb85176a1e344f78f7dfd87f39206b6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:45:52.026497 21573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eb85176a1e344f78f7dfd87f39206b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb85176a1e344f78f7dfd87f39206b6" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 45319 } }
I20260926 16:45:52.027374 21573 leader_election.cc:302] T 00000000000000000000000000000000 P 7eb85176a1e344f78f7dfd87f39206b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7eb85176a1e344f78f7dfd87f39206b6; no voters:
I20260926 16:45:52.028239 21573 leader_election.cc:288] T 00000000000000000000000000000000 P 7eb85176a1e344f78f7dfd87f39206b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:45:52.028465 21578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7eb85176a1e344f78f7dfd87f39206b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:45:52.029390 21578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7eb85176a1e344f78f7dfd87f39206b6 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb85176a1e344f78f7dfd87f39206b6, State: Running, Role: LEADER
I20260926 16:45:52.030049 21578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eb85176a1e344f78f7dfd87f39206b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb85176a1e344f78f7dfd87f39206b6" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 45319 } }
I20260926 16:45:52.031008 21573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eb85176a1e344f78f7dfd87f39206b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:45:52.033560 21580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb85176a1e344f78f7dfd87f39206b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eb85176a1e344f78f7dfd87f39206b6. Latest consensus state: current_term: 1 leader_uuid: "7eb85176a1e344f78f7dfd87f39206b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb85176a1e344f78f7dfd87f39206b6" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 45319 } } }
I20260926 16:45:52.033972 21580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb85176a1e344f78f7dfd87f39206b6 [sys.catalog]: This master's current role is: LEADER
I20260926 16:45:52.034722 21579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eb85176a1e344f78f7dfd87f39206b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7eb85176a1e344f78f7dfd87f39206b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eb85176a1e344f78f7dfd87f39206b6" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 45319 } } }
I20260926 16:45:52.035060 21579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eb85176a1e344f78f7dfd87f39206b6 [sys.catalog]: This master's current role is: LEADER
I20260926 16:45:52.035455 21584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:45:52.043430 21584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:45:52.046350 21573 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 16:45:52.063961 21584 catalog_manager.cc:1347] Generated new cluster ID: a3ef854ccf274431b9f8eabcdf3899cf
I20260926 16:45:52.064188 21584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:45:52.113301 21584 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:45:52.115919 21584 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:45:52.135190 21584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7eb85176a1e344f78f7dfd87f39206b6: Generated new TSK 0
I20260926 16:45:52.136646 21584 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
I20260926 16:45:52.988901 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.238.212:35827/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:0
--local_ip_for_outbound_sockets=127.9.238.193
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:45319
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:34771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:45:53 dist-test-slave-wt2j krb5kdc[21062](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790441153, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 16:45:53.537417 21618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:45:53.537832 21618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:45:53.538028 21618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:45:53.547000 21618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:45:53.547163 21618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:45:53.547387 21618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:45:53.560177 21618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:34771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.238.212:35827/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:45319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:45:53.562117 21618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:45:53.565418 21618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:45:53.596871 21628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:45:53.599107 21629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:45:53.607028 21618 server_base.cc:1034] running on GCE node
W20260926 16:45:53.607493 21631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:45:53.608798 21618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:45:53.610688 21618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:45:53.612175 21618 hybrid_clock.cc:648] HybridClock initialized: now 1790441153612040 us; error 130 us; skew 500 ppm
I20260926 16:45:53.612772 21618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:45:53 dist-test-slave-wt2j krb5kdc[21062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441153, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:45:53.618275 21618 init.cc:377] Logged in from keytab as kudu/127.9.238.193@KRBTEST.COM (short username kudu)
I20260926 16:45:53.631780 21618 webserver.cc:460] Webserver started at http://127.9.238.193:43239/ using document root <none> and password file <none>
I20260926 16:45:53.633294 21618 fs_manager.cc:359] Metadata directory not provided
I20260926 16:45:53.633495 21618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:45:53.633989 21618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:45:53 dist-test-slave-wt2j krb5kdc[21062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441153, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for HTTP/127.9.238.212@KRBTEST.COM
I20260926 16:45:53.662024 21618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-0/data/instance:
uuid: "ea4ae58d4e59494c9eb3bcf1ef7021e7"
format_stamp: "Formatted at 2026-09-26 16:45:53 on dist-test-slave-wt2j"
server_key: "ce2ecc3f18f6e509aa8cd8a00bdf583e"
server_key_iv: "3a1bbbd228a8f0c0edba08568cbc1649"
server_key_version: "kuduclusterkey@0"
I20260926 16:45:53.663578 21618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance:
uuid: "ea4ae58d4e59494c9eb3bcf1ef7021e7"
format_stamp: "Formatted at 2026-09-26 16:45:53 on dist-test-slave-wt2j"
server_key: "ce2ecc3f18f6e509aa8cd8a00bdf583e"
server_key_iv: "3a1bbbd228a8f0c0edba08568cbc1649"
server_key_version: "kuduclusterkey@0"
I20260926 16:45:53.681867 21618 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.009s sys 0.006s
I20260926 16:45:53.728726 21638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:53.732035 21618 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.008s sys 0.009s
I20260926 16:45:53.732591 21618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "ea4ae58d4e59494c9eb3bcf1ef7021e7"
format_stamp: "Formatted at 2026-09-26 16:45:53 on dist-test-slave-wt2j"
server_key: "ce2ecc3f18f6e509aa8cd8a00bdf583e"
server_key_iv: "3a1bbbd228a8f0c0edba08568cbc1649"
server_key_version: "kuduclusterkey@0"
I20260926 16:45:53.733481 21618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 26 16:45:53 dist-test-slave-wt2j krb5kdc[21062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441153, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for HTTP/127.9.238.212@KRBTEST.COM
I20260926 16:45:53.808231 21618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:45:53.819865 21618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:45:53.820536 21618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:45:53.824335 21618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:45:53.826823 21618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:45:53.827031 21618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 16:45:53.827306 21618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:45:53.827585 21618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:54.013276 21618 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:36215
I20260926 16:45:54.013301 21750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:36215 every 8 connection(s)
I20260926 16:45:54.015352 21618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:45:54.016999 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 21618
I20260926 16:45:54.017236 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance
I20260926 16:45:54.018167 10171 external_mini_cluster.cc:1428] Setting key e404e61532dccf2380a6f28a21f57214
Sep 26 16:45:54 dist-test-slave-wt2j krb5kdc[21062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441153, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.238.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.194@KRBTEST.COM" created.
I20260926 16:45:54.079355 21755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:54.038584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:35757 (local address 127.9.238.254:45319)
0926 16:45:54.044702 (+ 6118us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:54.044707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:54.045449 (+ 742us) server_negotiation.cc:407] Connection header received
0926 16:45:54.048603 (+ 3154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:54.048613 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:54.048751 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:54.048933 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:45:54.052028 (+ 3095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:54.053392 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:54.056382 (+ 2990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:54.057092 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:54.057210 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:54.063787 (+ 6577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:45:54.063799 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:45:54.063804 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:45:54.063894 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:45:54.069285 (+ 5391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:54.070646 (+ 1361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:54.070655 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:54.070659 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:54.070919 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:54.071537 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:54.071542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:54.071570 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:54.071987 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:45:54.072127 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:54.078535 (+ 6408us) server_negotiation.cc:299] Negotiation successful
0926 16:45:54.078886 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5972,"thread_start_us":5843,"threads_started":1}
I20260926 16:45:54.081094 21754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:54.037653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:45319 (local address 127.9.238.193:35757)
0926 16:45:54.045242 (+ 7589us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:54.045296 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 16:45:54.048109 (+ 2813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:45:54.049635 (+ 1526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:45:54.049654 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:45:54.050503 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:45:54.051699 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:54.051737 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:54.053674 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:54.053680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:54.056155 (+ 2475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:54.056174 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:54.057684 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:54.057692 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:54.058070 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:54.059585 (+ 1515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:45:54.059661 (+ 76us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:45:54.063334 (+ 3673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:45:54.069599 (+ 6265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:45:54.069613 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:45:54.069642 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:45:54.070446 (+ 804us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:45:54.071087 (+ 641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:45:54.071093 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:45:54.071097 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:45:54.071398 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:45:54.072235 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:45:54.072246 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:45:54.072829 (+ 583us) client_negotiation.cc:769] Sending connection context
0926 16:45:54.078245 (+ 5416us) client_negotiation.cc:240] Negotiation successful
0926 16:45:54.079065 (+ 820us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7246,"thread_start_us":6989,"threads_started":1}
I20260926 16:45:54.083489 21751 heartbeater.cc:347] Connected to a master server at 127.9.238.254:45319
I20260926 16:45:54.083950 21751 heartbeater.cc:464] Registering TS with master...
I20260926 16:45:54.085037 21751 heartbeater.cc:511] Master 127.9.238.254:45319 requested a full tablet report, sending...
I20260926 16:45:54.088567 21538 ts_manager.cc:194] Registered new tserver with Master: ea4ae58d4e59494c9eb3bcf1ef7021e7 (127.9.238.193:36215)
I20260926 16:45:54.091463 21538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.193@KRBTEST.COM'} at 127.9.238.193:35757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
I20260926 16:45:54.112569 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.238.212:35827/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:0
--local_ip_for_outbound_sockets=127.9.238.194
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:45319
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:34771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:45:54 dist-test-slave-wt2j krb5kdc[21062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441153, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for HTTP/127.9.238.212@KRBTEST.COM
W20260926 16:45:54.654424 21757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:45:54.654733 21757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:45:54.654863 21757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:45:54.662547 21757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:45:54.662775 21757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:45:54.662985 21757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:45:54.673693 21757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:34771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.238.212:35827/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:45319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:45:54.675115 21757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:45:54.676453 21757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:45:54.693890 21771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:45:54.694283 21772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:45:54.695850 21757 server_base.cc:1034] running on GCE node
W20260926 16:45:54.696058 21774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:45:54.697029 21757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:45:54.698419 21757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:45:54.699707 21757 hybrid_clock.cc:648] HybridClock initialized: now 1790441154699597 us; error 71 us; skew 500 ppm
I20260926 16:45:54.700160 21757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:45:54 dist-test-slave-wt2j krb5kdc[21062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441154, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:45:54.704216 21757 init.cc:377] Logged in from keytab as kudu/127.9.238.194@KRBTEST.COM (short username kudu)
I20260926 16:45:54.706348 21757 webserver.cc:460] Webserver started at http://127.9.238.194:42749/ using document root <none> and password file <none>
I20260926 16:45:54.707207 21757 fs_manager.cc:359] Metadata directory not provided
I20260926 16:45:54.707358 21757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:45:54.707800 21757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:45:54 dist-test-slave-wt2j krb5kdc[21062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441154, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for HTTP/127.9.238.212@KRBTEST.COM
I20260926 16:45:54.729902 21757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-1/data/instance:
uuid: "adf8ab33231d4dd8ac6eabccb177dfb5"
format_stamp: "Formatted at 2026-09-26 16:45:54 on dist-test-slave-wt2j"
server_key: "b3328e0af0566510cf83ab7e30cc8f22"
server_key_iv: "9d8e546c8cc3ceab50b81af9649c88a5"
server_key_version: "kuduclusterkey@0"
I20260926 16:45:54.731233 21757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance:
uuid: "adf8ab33231d4dd8ac6eabccb177dfb5"
format_stamp: "Formatted at 2026-09-26 16:45:54 on dist-test-slave-wt2j"
server_key: "b3328e0af0566510cf83ab7e30cc8f22"
server_key_iv: "9d8e546c8cc3ceab50b81af9649c88a5"
server_key_version: "kuduclusterkey@0"
I20260926 16:45:54.741312 21757 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.007s sys 0.001s
I20260926 16:45:54.768307 21781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:54.770813 21757 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260926 16:45:54.771083 21757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "adf8ab33231d4dd8ac6eabccb177dfb5"
format_stamp: "Formatted at 2026-09-26 16:45:54 on dist-test-slave-wt2j"
server_key: "b3328e0af0566510cf83ab7e30cc8f22"
server_key_iv: "9d8e546c8cc3ceab50b81af9649c88a5"
server_key_version: "kuduclusterkey@0"
I20260926 16:45:54.771418 21757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:45:54.802901 21757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:45:54.809847 21757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:45:54.810276 21757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:45:54.811885 21757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:45:54.814267 21757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:45:54.814363 21757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:54.814494 21757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:45:54.814553 21757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:54.880297 21757 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:42525
I20260926 16:45:54.880323 21893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:42525 every 8 connection(s)
I20260926 16:45:54.882822 21757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:45:54.889338 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 21757
I20260926 16:45:54.889541 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance
I20260926 16:45:54.890116 10171 external_mini_cluster.cc:1428] Setting key 9918a420da7c4f3ae5a981541ae6a508
Sep 26 16:45:54 dist-test-slave-wt2j krb5kdc[21062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441154, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.238.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.195@KRBTEST.COM" created.
I20260926 16:45:54.930856 21897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:54.888432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:45319 (local address 127.9.238.194:55753)
0926 16:45:54.891322 (+ 2890us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:54.891418 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0926 16:45:54.895729 (+ 4311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:45:54.905810 (+ 10081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:45:54.905833 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:45:54.906759 (+ 926us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:45:54.908031 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:54.908072 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:54.910194 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:54.910212 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:54.912839 (+ 2627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:54.912880 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:54.914553 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:54.914562 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:54.915016 (+ 454us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:54.916274 (+ 1258us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:45:54.916319 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:45:54.919948 (+ 3629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:45:54.925461 (+ 5513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:45:54.925474 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:45:54.925496 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:45:54.926265 (+ 769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:45:54.926843 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:45:54.926847 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:45:54.926851 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:45:54.927117 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:45:54.927961 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:45:54.927970 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:45:54.928442 (+ 472us) client_negotiation.cc:769] Sending connection context
0926 16:45:54.928763 (+ 321us) client_negotiation.cc:240] Negotiation successful
0926 16:45:54.929369 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2263,"thread_start_us":3169,"threads_started":1}
I20260926 16:45:54.932433 21896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:54.887790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:55753 (local address 127.9.238.254:45319)
0926 16:45:54.904739 (+ 16949us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:54.904746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:54.904775 (+ 29us) server_negotiation.cc:407] Connection header received
0926 16:45:54.904859 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:54.904866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:54.904982 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:54.905158 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:45:54.908452 (+ 3294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:54.909886 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:54.913157 (+ 3271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:54.913920 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:54.914083 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:54.920388 (+ 6305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:45:54.920399 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:45:54.920404 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:45:54.920462 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:45:54.925161 (+ 4699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:54.926499 (+ 1338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:54.926507 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:54.926512 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:54.926703 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:54.927258 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:54.927263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:54.927266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:54.927715 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:45:54.927874 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:54.931766 (+ 3892us) server_negotiation.cc:299] Negotiation successful
0926 16:45:54.932077 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16765,"thread_start_us":16628,"threads_started":1}
I20260926 16:45:54.936537 21894 heartbeater.cc:347] Connected to a master server at 127.9.238.254:45319
I20260926 16:45:54.936964 21894 heartbeater.cc:464] Registering TS with master...
I20260926 16:45:54.938141 21894 heartbeater.cc:511] Master 127.9.238.254:45319 requested a full tablet report, sending...
I20260926 16:45:54.940214 21538 ts_manager.cc:194] Registered new tserver with Master: adf8ab33231d4dd8ac6eabccb177dfb5 (127.9.238.194:42525)
I20260926 16:45:54.941778 21538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.194@KRBTEST.COM'} at 127.9.238.194:55753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
I20260926 16:45:54.975224 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.238.212:35827/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:0
--local_ip_for_outbound_sockets=127.9.238.195
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:45319
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:34771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:45:55.098106 21751 heartbeater.cc:503] Master 127.9.238.254:45319 was elected leader, sending a full tablet report...
W20260926 16:45:55.367791 21900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:45:55.368134 21900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:45:55.368229 21900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:45:55.375952 21900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:45:55.376080 21900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:45:55.376200 21900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:45:55.387672 21900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:34771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.238.212:35827/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:45319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:45:55.389106 21900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:45:55.390336 21900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:45:55.407271 21908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:45:55.407222 21906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:45:55.407320 21905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:45:55.408869 21900 server_base.cc:1034] running on GCE node
I20260926 16:45:55.409562 21900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:45:55.410702 21900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:45:55.412365 21900 hybrid_clock.cc:648] HybridClock initialized: now 1790441155411897 us; error 42 us; skew 500 ppm
I20260926 16:45:55.412724 21900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:45:55 dist-test-slave-wt2j krb5kdc[21062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441155, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:45:55.416465 21900 init.cc:377] Logged in from keytab as kudu/127.9.238.195@KRBTEST.COM (short username kudu)
I20260926 16:45:55.418332 21900 webserver.cc:460] Webserver started at http://127.9.238.195:41311/ using document root <none> and password file <none>
I20260926 16:45:55.419081 21900 fs_manager.cc:359] Metadata directory not provided
I20260926 16:45:55.419229 21900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:45:55.419538 21900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:45:55 dist-test-slave-wt2j krb5kdc[21062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441155, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for HTTP/127.9.238.212@KRBTEST.COM
I20260926 16:45:55.437045 21900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-2/data/instance:
uuid: "6981b6d0a4cf4606999908a3f8401924"
format_stamp: "Formatted at 2026-09-26 16:45:55 on dist-test-slave-wt2j"
server_key: "b3ef33a871c4aacc322a72a06f8291d9"
server_key_iv: "26618f8ff4b721571a5bd6a656526282"
server_key_version: "kuduclusterkey@0"
I20260926 16:45:55.438122 21900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance:
uuid: "6981b6d0a4cf4606999908a3f8401924"
format_stamp: "Formatted at 2026-09-26 16:45:55 on dist-test-slave-wt2j"
server_key: "b3ef33a871c4aacc322a72a06f8291d9"
server_key_iv: "26618f8ff4b721571a5bd6a656526282"
server_key_version: "kuduclusterkey@0"
I20260926 16:45:55.445566 21900 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260926 16:45:55.468222 21915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:55.470374 21900 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260926 16:45:55.470642 21900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "6981b6d0a4cf4606999908a3f8401924"
format_stamp: "Formatted at 2026-09-26 16:45:55 on dist-test-slave-wt2j"
server_key: "b3ef33a871c4aacc322a72a06f8291d9"
server_key_iv: "26618f8ff4b721571a5bd6a656526282"
server_key_version: "kuduclusterkey@0"
I20260926 16:45:55.470942 21900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:45:55.491707 21900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:45:55.496433 21900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:45:55.496781 21900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:45:55.497951 21900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:45:55.499524 21900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:45:55.499670 21900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260926 16:45:55.499838 21900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:45:55.499893 21900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:45:55.545228 21900 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:43041
I20260926 16:45:55.545279 22027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:43041 every 8 connection(s)
I20260926 16:45:55.546672 21900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:45:55.554116 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 21900
I20260926 16:45:55.554312 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance
I20260926 16:45:55.554852 10171 external_mini_cluster.cc:1428] Setting key 99c519825bee80e61800588a45a8bbf3
Sep 26 16:45:55 dist-test-slave-wt2j krb5kdc[21062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441155, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:45:55.571920 22030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:55.550207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:53793 (local address 127.9.238.254:45319)
0926 16:45:55.550895 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:55.550903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:55.552294 (+ 1391us) server_negotiation.cc:407] Connection header received
0926 16:45:55.554459 (+ 2165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:55.554466 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:55.554560 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:55.554714 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:45:55.556767 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.557799 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.559715 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.560322 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.560438 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:55.564481 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:45:55.564488 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:45:55.564492 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:45:55.564540 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:45:55.567754 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:55.568654 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:55.568659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:55.568661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:55.568809 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:55.569441 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:55.569448 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:55.569452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:55.569960 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:45:55.570168 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:55.571126 (+ 958us) server_negotiation.cc:299] Negotiation successful
0926 16:45:55.571389 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":300,"threads_started":1}
I20260926 16:45:55.572803 22031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:55.551152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:45319 (local address 127.9.238.195:53793)
0926 16:45:55.552090 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:55.552138 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 16:45:55.554037 (+ 1899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:45:55.555029 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:45:55.555042 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:45:55.555743 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:45:55.556540 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:55.556570 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.557957 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.557961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:55.559473 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:55.559492 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.560531 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.560536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:55.561176 (+ 640us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:55.561909 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:45:55.561932 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:45:55.564126 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:45:55.567939 (+ 3813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:45:55.567947 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:45:55.567962 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:45:55.568515 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:45:55.568926 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:45:55.568932 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:45:55.568938 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:45:55.569224 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:45:55.570119 (+ 895us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:45:55.570125 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:45:55.570648 (+ 523us) client_negotiation.cc:769] Sending connection context
0926 16:45:55.570976 (+ 328us) client_negotiation.cc:240] Negotiation successful
0926 16:45:55.571480 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":408,"threads_started":1}
I20260926 16:45:55.574719 22028 heartbeater.cc:347] Connected to a master server at 127.9.238.254:45319
I20260926 16:45:55.575157 22028 heartbeater.cc:464] Registering TS with master...
I20260926 16:45:55.576359 22028 heartbeater.cc:511] Master 127.9.238.254:45319 requested a full tablet report, sending...
I20260926 16:45:55.578504 21538 ts_manager.cc:194] Registered new tserver with Master: 6981b6d0a4cf4606999908a3f8401924 (127.9.238.195:43041)
I20260926 16:45:55.579612 21538 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.195@KRBTEST.COM'} at 127.9.238.195:53793
I20260926 16:45:55.585080 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:45:55.606644 22030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:55.590755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32828 (local address 127.9.238.254:45319)
0926 16:45:55.591070 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:55.591078 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:55.591285 (+ 207us) server_negotiation.cc:407] Connection header received
0926 16:45:55.591605 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:55.591611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:55.591707 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:55.591826 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:45:55.593641 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.594434 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.596704 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.597168 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.597315 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:55.600410 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:45:55.600417 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:45:55.600420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:45:55.600463 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:45:55.603532 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:55.604424 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:55.604432 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:55.604436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:55.604589 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:55.604966 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:55.604973 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:55.604977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:55.605295 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:45:55.605564 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:55.605981 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 16:45:55.606242 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260926 16:45:55.611539 21538 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:45:55.620286 21538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:45:55.664361 22041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:55.637400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:42525 (local address 127.0.0.1:35126)
0926 16:45:55.651095 (+ 13695us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:55.651130 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 16:45:55.651347 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:45:55.654338 (+ 2991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:45:55.654347 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:45:55.654387 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:45:55.656221 (+ 1834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:55.656240 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.659113 (+ 2873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.659118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:55.661566 (+ 2448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:55.661578 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.663063 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.663069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:55.663267 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:55.663326 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 16:45:55.663507 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 16:45:55.663784 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13457,"mutex_wait_us":117,"thread_start_us":14310,"threads_started":1}
I20260926 16:45:55.664384 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:55.637252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:43041 (local address 127.0.0.1:40190)
0926 16:45:55.648283 (+ 11031us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:55.648396 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0926 16:45:55.648812 (+ 416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:45:55.654804 (+ 5992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:45:55.654813 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:45:55.654846 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:45:55.656344 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:55.656359 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.658529 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.658534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:55.661431 (+ 2897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:55.661450 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.663483 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.663493 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:55.663692 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:55.663723 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 16:45:55.663801 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 16:45:55.663918 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10437,"mutex_wait_us":371,"thread_start_us":13372,"threads_started":1}
I20260926 16:45:55.664915 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:55.638856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54844 (local address 127.9.238.193:36215)
0926 16:45:55.653046 (+ 14190us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:55.653059 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:55.653098 (+ 39us) server_negotiation.cc:407] Connection header received
0926 16:45:55.653253 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:55.653264 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:55.653635 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:55.654743 (+ 1108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:45:55.656081 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.657721 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.661329 (+ 3608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.663166 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.663335 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:55.663791 (+ 456us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:55.664299 (+ 508us) server_negotiation.cc:299] Negotiation successful
0926 16:45:55.664497 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13935,"thread_start_us":13705,"threads_started":1}
I20260926 16:45:55.665467 22042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:55.637693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:36215 (local address 127.0.0.1:54844)
0926 16:45:55.651979 (+ 14286us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:55.652006 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 16:45:55.652153 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:45:55.653999 (+ 1846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:45:55.654009 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:45:55.654049 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:45:55.655767 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:55.655785 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.657871 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.657878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:55.661072 (+ 3194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:55.661091 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.663905 (+ 2814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.663911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:55.664062 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:55.664091 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 16:45:55.664234 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 16:45:55.664968 (+ 734us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14163,"mutex_wait_us":222,"thread_start_us":11574,"threads_started":1}
I20260926 16:45:55.666905 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:55.638594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40190 (local address 127.9.238.195:43041)
0926 16:45:55.654138 (+ 15544us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:55.654150 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:55.654194 (+ 44us) server_negotiation.cc:407] Connection header received
0926 16:45:55.654310 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:55.654318 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:55.654556 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:55.654832 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:45:55.656634 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.658351 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.661637 (+ 3286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.663203 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.663389 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:55.663657 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:55.665412 (+ 1755us) server_negotiation.cc:299] Negotiation successful
0926 16:45:55.666497 (+ 1085us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15321,"thread_start_us":15078,"threads_started":1}
I20260926 16:45:55.666996 22044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:55.638065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35126 (local address 127.9.238.194:42525)
0926 16:45:55.653472 (+ 15407us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:55.653488 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:55.653540 (+ 52us) server_negotiation.cc:407] Connection header received
0926 16:45:55.653696 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:55.653706 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:55.654067 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:55.654332 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:45:55.656566 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.658351 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.661744 (+ 3393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.662829 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.663004 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:55.663265 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:55.666382 (+ 3117us) server_negotiation.cc:299] Negotiation successful
0926 16:45:55.666591 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15151,"thread_start_us":15008,"threads_started":1}
I20260926 16:45:55.670775 21829 tablet_service.cc:1467] Processing CreateTablet for tablet 8ee6889a458743b9ab4bb495fa75dccc (DEFAULT_TABLE table=test-table [id=4a5b5e73f2744ce297420a5aed5d6ec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:45:55.670676 21686 tablet_service.cc:1467] Processing CreateTablet for tablet 8ee6889a458743b9ab4bb495fa75dccc (DEFAULT_TABLE table=test-table [id=4a5b5e73f2744ce297420a5aed5d6ec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:45:55.670917 21963 tablet_service.cc:1467] Processing CreateTablet for tablet 8ee6889a458743b9ab4bb495fa75dccc (DEFAULT_TABLE table=test-table [id=4a5b5e73f2744ce297420a5aed5d6ec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:45:55.672847 21829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ee6889a458743b9ab4bb495fa75dccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:45:55.672878 21686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ee6889a458743b9ab4bb495fa75dccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:45:55.673132 21963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ee6889a458743b9ab4bb495fa75dccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:45:55.688823 22047 tablet_bootstrap.cc:492] T 8ee6889a458743b9ab4bb495fa75dccc P 6981b6d0a4cf4606999908a3f8401924: Bootstrap starting.
I20260926 16:45:55.692322 22049 tablet_bootstrap.cc:492] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7: Bootstrap starting.
I20260926 16:45:55.694567 22047 tablet_bootstrap.cc:654] T 8ee6889a458743b9ab4bb495fa75dccc P 6981b6d0a4cf4606999908a3f8401924: Neither blocks nor log segments found. Creating new log.
I20260926 16:45:55.695820 22048 tablet_bootstrap.cc:492] T 8ee6889a458743b9ab4bb495fa75dccc P adf8ab33231d4dd8ac6eabccb177dfb5: Bootstrap starting.
I20260926 16:45:55.698099 22047 log.cc:824] T 8ee6889a458743b9ab4bb495fa75dccc P 6981b6d0a4cf4606999908a3f8401924: Log is configured to *not* fsync() on all Append() calls
I20260926 16:45:55.699987 22048 tablet_bootstrap.cc:654] T 8ee6889a458743b9ab4bb495fa75dccc P adf8ab33231d4dd8ac6eabccb177dfb5: Neither blocks nor log segments found. Creating new log.
I20260926 16:45:55.700637 22049 tablet_bootstrap.cc:654] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7: Neither blocks nor log segments found. Creating new log.
I20260926 16:45:55.701447 22048 log.cc:824] T 8ee6889a458743b9ab4bb495fa75dccc P adf8ab33231d4dd8ac6eabccb177dfb5: Log is configured to *not* fsync() on all Append() calls
I20260926 16:45:55.701984 22049 log.cc:824] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7: Log is configured to *not* fsync() on all Append() calls
I20260926 16:45:55.702183 22047 tablet_bootstrap.cc:492] T 8ee6889a458743b9ab4bb495fa75dccc P 6981b6d0a4cf4606999908a3f8401924: No bootstrap required, opened a new log
I20260926 16:45:55.702534 22047 ts_tablet_manager.cc:1397] T 8ee6889a458743b9ab4bb495fa75dccc P 6981b6d0a4cf4606999908a3f8401924: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.009s
I20260926 16:45:55.704986 22048 tablet_bootstrap.cc:492] T 8ee6889a458743b9ab4bb495fa75dccc P adf8ab33231d4dd8ac6eabccb177dfb5: No bootstrap required, opened a new log
I20260926 16:45:55.705307 22048 ts_tablet_manager.cc:1397] T 8ee6889a458743b9ab4bb495fa75dccc P adf8ab33231d4dd8ac6eabccb177dfb5: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260926 16:45:55.705647 22049 tablet_bootstrap.cc:492] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7: No bootstrap required, opened a new log
I20260926 16:45:55.705960 22049 ts_tablet_manager.cc:1397] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20260926 16:45:55.713980 22047 raft_consensus.cc:348] T 8ee6889a458743b9ab4bb495fa75dccc P 6981b6d0a4cf4606999908a3f8401924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6981b6d0a4cf4606999908a3f8401924" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 43041 } } peers { permanent_uuid: "adf8ab33231d4dd8ac6eabccb177dfb5" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42525 } } peers { permanent_uuid: "ea4ae58d4e59494c9eb3bcf1ef7021e7" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 36215 } }
I20260926 16:45:55.714473 22047 raft_consensus.cc:374] T 8ee6889a458743b9ab4bb495fa75dccc P 6981b6d0a4cf4606999908a3f8401924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:45:55.714578 22047 raft_consensus.cc:729] T 8ee6889a458743b9ab4bb495fa75dccc P 6981b6d0a4cf4606999908a3f8401924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6981b6d0a4cf4606999908a3f8401924, State: Initialized, Role: FOLLOWER
I20260926 16:45:55.714506 22048 raft_consensus.cc:348] T 8ee6889a458743b9ab4bb495fa75dccc P adf8ab33231d4dd8ac6eabccb177dfb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6981b6d0a4cf4606999908a3f8401924" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 43041 } } peers { permanent_uuid: "adf8ab33231d4dd8ac6eabccb177dfb5" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42525 } } peers { permanent_uuid: "ea4ae58d4e59494c9eb3bcf1ef7021e7" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 36215 } }
I20260926 16:45:55.715003 22048 raft_consensus.cc:374] T 8ee6889a458743b9ab4bb495fa75dccc P adf8ab33231d4dd8ac6eabccb177dfb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:45:55.715101 22048 raft_consensus.cc:729] T 8ee6889a458743b9ab4bb495fa75dccc P adf8ab33231d4dd8ac6eabccb177dfb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adf8ab33231d4dd8ac6eabccb177dfb5, State: Initialized, Role: FOLLOWER
I20260926 16:45:55.715303 22047 consensus_queue.cc:260] T 8ee6889a458743b9ab4bb495fa75dccc P 6981b6d0a4cf4606999908a3f8401924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6981b6d0a4cf4606999908a3f8401924" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 43041 } } peers { permanent_uuid: "adf8ab33231d4dd8ac6eabccb177dfb5" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42525 } } peers { permanent_uuid: "ea4ae58d4e59494c9eb3bcf1ef7021e7" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 36215 } }
I20260926 16:45:55.715870 22048 consensus_queue.cc:260] T 8ee6889a458743b9ab4bb495fa75dccc P adf8ab33231d4dd8ac6eabccb177dfb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6981b6d0a4cf4606999908a3f8401924" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 43041 } } peers { permanent_uuid: "adf8ab33231d4dd8ac6eabccb177dfb5" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42525 } } peers { permanent_uuid: "ea4ae58d4e59494c9eb3bcf1ef7021e7" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 36215 } }
I20260926 16:45:55.717602 22049 raft_consensus.cc:348] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6981b6d0a4cf4606999908a3f8401924" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 43041 } } peers { permanent_uuid: "adf8ab33231d4dd8ac6eabccb177dfb5" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42525 } } peers { permanent_uuid: "ea4ae58d4e59494c9eb3bcf1ef7021e7" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 36215 } }
I20260926 16:45:55.717995 22028 heartbeater.cc:503] Master 127.9.238.254:45319 was elected leader, sending a full tablet report...
I20260926 16:45:55.718067 22049 raft_consensus.cc:374] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:45:55.718192 22049 raft_consensus.cc:729] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea4ae58d4e59494c9eb3bcf1ef7021e7, State: Initialized, Role: FOLLOWER
I20260926 16:45:55.718717 21894 heartbeater.cc:503] Master 127.9.238.254:45319 was elected leader, sending a full tablet report...
I20260926 16:45:55.718870 22049 consensus_queue.cc:260] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6981b6d0a4cf4606999908a3f8401924" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 43041 } } peers { permanent_uuid: "adf8ab33231d4dd8ac6eabccb177dfb5" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42525 } } peers { permanent_uuid: "ea4ae58d4e59494c9eb3bcf1ef7021e7" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 36215 } }
I20260926 16:45:55.722018 22048 ts_tablet_manager.cc:1428] T 8ee6889a458743b9ab4bb495fa75dccc P adf8ab33231d4dd8ac6eabccb177dfb5: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260926 16:45:55.722061 22047 ts_tablet_manager.cc:1428] T 8ee6889a458743b9ab4bb495fa75dccc P 6981b6d0a4cf4606999908a3f8401924: Time spent starting tablet: real 0.019s user 0.016s sys 0.002s
I20260926 16:45:55.724718 22049 ts_tablet_manager.cc:1428] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7: Time spent starting tablet: real 0.019s user 0.020s sys 0.000s
W20260926 16:45:55.795948 21753 tablet.cc:2369] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:45:55.798385 22029 tablet.cc:2369] T 8ee6889a458743b9ab4bb495fa75dccc P 6981b6d0a4cf4606999908a3f8401924: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:45:55.894918 21895 tablet.cc:2369] T 8ee6889a458743b9ab4bb495fa75dccc P adf8ab33231d4dd8ac6eabccb177dfb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:45:55.962862 22055 raft_consensus.cc:482] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:45:55.963337 22055 raft_consensus.cc:504] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6981b6d0a4cf4606999908a3f8401924" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 43041 } } peers { permanent_uuid: "adf8ab33231d4dd8ac6eabccb177dfb5" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42525 } } peers { permanent_uuid: "ea4ae58d4e59494c9eb3bcf1ef7021e7" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 36215 } }
I20260926 16:45:55.965648 22055 leader_election.cc:288] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6981b6d0a4cf4606999908a3f8401924 (127.9.238.195:43041), adf8ab33231d4dd8ac6eabccb177dfb5 (127.9.238.194:42525)
I20260926 16:45:55.975196 22044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:55.966412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:54425 (local address 127.9.238.194:42525)
0926 16:45:55.966852 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:55.966859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:55.967956 (+ 1097us) server_negotiation.cc:407] Connection header received
0926 16:45:55.968293 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:55.968302 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:55.968392 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:55.968474 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:45:55.969195 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.970104 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.972556 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.973872 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.974091 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:55.974192 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:55.974673 (+ 481us) server_negotiation.cc:299] Negotiation successful
0926 16:45:55.974877 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229}
I20260926 16:45:55.975489 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:55.966220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:52491 (local address 127.9.238.195:43041)
0926 16:45:55.966541 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:55.966551 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:55.968120 (+ 1569us) server_negotiation.cc:407] Connection header received
0926 16:45:55.968339 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:55.968343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:55.968414 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:55.968544 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:45:55.969363 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.970130 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.973078 (+ 2948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.974403 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.974610 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:55.974688 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:55.975020 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 16:45:55.975159 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260926 16:45:55.975629 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:55.966277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:43041 (local address 127.9.238.193:52491)
0926 16:45:55.967969 (+ 1692us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:55.967988 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:45:55.968175 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:45:55.968593 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:45:55.968598 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:45:55.968623 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:45:55.969182 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:55.969195 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.970315 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.970319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:55.972885 (+ 2566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:55.972906 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.974652 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.974659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:55.974843 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:55.974873 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 16:45:55.974991 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 16:45:55.975173 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1513,"mutex_wait_us":22,"thread_start_us":1189,"threads_started":1}
I20260926 16:45:55.976331 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:55.966214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:42525 (local address 127.9.238.193:54425)
0926 16:45:55.967787 (+ 1573us) negotiation.cc:107] Waiting for socket to connect
0926 16:45:55.967807 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:45:55.968044 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:45:55.968557 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:45:55.968562 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:45:55.968591 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:45:55.969028 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:55.969038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.970261 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.970266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:55.972337 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:45:55.972350 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:45:55.974123 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:45:55.974130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:45:55.974433 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:55.974475 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 16:45:55.975865 (+ 1390us) client_negotiation.cc:240] Negotiation successful
0926 16:45:55.976005 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1419,"mutex_wait_us":43,"thread_start_us":1045,"threads_started":1}
I20260926 16:45:55.977948 21849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ee6889a458743b9ab4bb495fa75dccc" candidate_uuid: "ea4ae58d4e59494c9eb3bcf1ef7021e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adf8ab33231d4dd8ac6eabccb177dfb5" is_pre_election: true
I20260926 16:45:55.978086 21983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ee6889a458743b9ab4bb495fa75dccc" candidate_uuid: "ea4ae58d4e59494c9eb3bcf1ef7021e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6981b6d0a4cf4606999908a3f8401924" is_pre_election: true
I20260926 16:45:55.978637 21849 raft_consensus.cc:2454] T 8ee6889a458743b9ab4bb495fa75dccc P adf8ab33231d4dd8ac6eabccb177dfb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea4ae58d4e59494c9eb3bcf1ef7021e7 in term 0.
I20260926 16:45:55.978772 21983 raft_consensus.cc:2454] T 8ee6889a458743b9ab4bb495fa75dccc P 6981b6d0a4cf4606999908a3f8401924 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea4ae58d4e59494c9eb3bcf1ef7021e7 in term 0.
I20260926 16:45:55.979393 21642 leader_election.cc:302] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adf8ab33231d4dd8ac6eabccb177dfb5, ea4ae58d4e59494c9eb3bcf1ef7021e7; no voters:
I20260926 16:45:55.979892 22055 raft_consensus.cc:2781] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:45:55.980057 22055 raft_consensus.cc:482] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:45:55.980177 22055 raft_consensus.cc:3037] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:45:55.982623 22055 raft_consensus.cc:504] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6981b6d0a4cf4606999908a3f8401924" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 43041 } } peers { permanent_uuid: "adf8ab33231d4dd8ac6eabccb177dfb5" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42525 } } peers { permanent_uuid: "ea4ae58d4e59494c9eb3bcf1ef7021e7" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 36215 } }
I20260926 16:45:55.983335 22055 leader_election.cc:288] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7 [CANDIDATE]: Term 1 election: Requested vote from peers 6981b6d0a4cf4606999908a3f8401924 (127.9.238.195:43041), adf8ab33231d4dd8ac6eabccb177dfb5 (127.9.238.194:42525)
I20260926 16:45:55.983938 21849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ee6889a458743b9ab4bb495fa75dccc" candidate_uuid: "ea4ae58d4e59494c9eb3bcf1ef7021e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adf8ab33231d4dd8ac6eabccb177dfb5"
I20260926 16:45:55.984053 21983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ee6889a458743b9ab4bb495fa75dccc" candidate_uuid: "ea4ae58d4e59494c9eb3bcf1ef7021e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6981b6d0a4cf4606999908a3f8401924"
I20260926 16:45:55.984194 21849 raft_consensus.cc:3037] T 8ee6889a458743b9ab4bb495fa75dccc P adf8ab33231d4dd8ac6eabccb177dfb5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:45:55.984305 21983 raft_consensus.cc:3037] T 8ee6889a458743b9ab4bb495fa75dccc P 6981b6d0a4cf4606999908a3f8401924 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:45:55.987917 21983 raft_consensus.cc:2454] T 8ee6889a458743b9ab4bb495fa75dccc P 6981b6d0a4cf4606999908a3f8401924 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea4ae58d4e59494c9eb3bcf1ef7021e7 in term 1.
I20260926 16:45:55.988225 21849 raft_consensus.cc:2454] T 8ee6889a458743b9ab4bb495fa75dccc P adf8ab33231d4dd8ac6eabccb177dfb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea4ae58d4e59494c9eb3bcf1ef7021e7 in term 1.
I20260926 16:45:55.988624 21641 leader_election.cc:302] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6981b6d0a4cf4606999908a3f8401924, ea4ae58d4e59494c9eb3bcf1ef7021e7; no voters:
I20260926 16:45:55.989053 22055 raft_consensus.cc:2781] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:45:55.989621 22055 raft_consensus.cc:686] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7 [term 1 LEADER]: Becoming Leader. State: Replica: ea4ae58d4e59494c9eb3bcf1ef7021e7, State: Running, Role: LEADER
I20260926 16:45:55.990238 22055 consensus_queue.cc:237] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6981b6d0a4cf4606999908a3f8401924" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 43041 } } peers { permanent_uuid: "adf8ab33231d4dd8ac6eabccb177dfb5" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42525 } } peers { permanent_uuid: "ea4ae58d4e59494c9eb3bcf1ef7021e7" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 36215 } }
I20260926 16:45:55.996610 21538 catalog_manager.cc:5495] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea4ae58d4e59494c9eb3bcf1ef7021e7 (127.9.238.193). New cstate: current_term: 1 leader_uuid: "ea4ae58d4e59494c9eb3bcf1ef7021e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6981b6d0a4cf4606999908a3f8401924" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 43041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adf8ab33231d4dd8ac6eabccb177dfb5" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea4ae58d4e59494c9eb3bcf1ef7021e7" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 36215 } health_report { overall_health: HEALTHY } } }
I20260926 16:45:56.041388 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:56.035692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54852 (local address 127.9.238.193:36215)
0926 16:45:56.035991 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:56.035996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:56.036020 (+ 24us) server_negotiation.cc:407] Connection header received
0926 16:45:56.036117 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:56.036120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:56.036200 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:56.036399 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:45:56.036774 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:56.037520 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:56.038895 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:56.039633 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:56.039761 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:56.040269 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:45:56.040799 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:45:56.040963 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:56.041031 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 16:45:56.041113 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 16:45:56.052539 21983 raft_consensus.cc:1261] T 8ee6889a458743b9ab4bb495fa75dccc P 6981b6d0a4cf4606999908a3f8401924 [term 1 FOLLOWER]: Refusing update from remote peer ea4ae58d4e59494c9eb3bcf1ef7021e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:45:56.053194 21849 raft_consensus.cc:1261] T 8ee6889a458743b9ab4bb495fa75dccc P adf8ab33231d4dd8ac6eabccb177dfb5 [term 1 FOLLOWER]: Refusing update from remote peer ea4ae58d4e59494c9eb3bcf1ef7021e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:45:56.053555 22055 consensus_queue.cc:1035] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6981b6d0a4cf4606999908a3f8401924" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 43041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:45:56.054243 22058 consensus_queue.cc:1035] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adf8ab33231d4dd8ac6eabccb177dfb5" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:45:56.068480 22061 mvcc.cc:205] Tried to move back new op lower bound from 7333646975177764864 to 7333646974943178752. Current Snapshot: MvccSnapshot[applied={T|T < 7333646975177764864}]
I20260926 16:45:56.070544 22063 mvcc.cc:205] Tried to move back new op lower bound from 7333646975177764864 to 7333646974943178752. Current Snapshot: MvccSnapshot[applied={T|T < 7333646975177764864}]
I20260926 16:45:56.088443 21538 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:45:56.088833 21538 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:45:56.094262 21538 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7eb85176a1e344f78f7dfd87f39206b6: Sending DeleteTablet for 3 replicas of tablet 8ee6889a458743b9ab4bb495fa75dccc
I20260926 16:45:56.095728 21963 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ee6889a458743b9ab4bb495fa75dccc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:45:56 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:40190
I20260926 16:45:56.095965 21829 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ee6889a458743b9ab4bb495fa75dccc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:45:56 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:35126
I20260926 16:45:56.095867 21686 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ee6889a458743b9ab4bb495fa75dccc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:45:56 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:54844
I20260926 16:45:56.099494 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 21618
I20260926 16:45:56.108155 22072 tablet_replica.cc:330] T 8ee6889a458743b9ab4bb495fa75dccc P adf8ab33231d4dd8ac6eabccb177dfb5: stopping tablet replica
I20260926 16:45:56.108283 22071 tablet_replica.cc:330] T 8ee6889a458743b9ab4bb495fa75dccc P 6981b6d0a4cf4606999908a3f8401924: stopping tablet replica
I20260926 16:45:56.108656 22073 tablet_replica.cc:330] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7: stopping tablet replica
I20260926 16:45:56.108711 22071 raft_consensus.cc:2229] T 8ee6889a458743b9ab4bb495fa75dccc P 6981b6d0a4cf4606999908a3f8401924 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:45:56.108716 22072 raft_consensus.cc:2229] T 8ee6889a458743b9ab4bb495fa75dccc P adf8ab33231d4dd8ac6eabccb177dfb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:45:56.109243 22071 raft_consensus.cc:2258] T 8ee6889a458743b9ab4bb495fa75dccc P 6981b6d0a4cf4606999908a3f8401924 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:45:56.109243 22072 raft_consensus.cc:2258] T 8ee6889a458743b9ab4bb495fa75dccc P adf8ab33231d4dd8ac6eabccb177dfb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:45:56.109575 22073 raft_consensus.cc:2229] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:45:56.111001 22073 raft_consensus.cc:2258] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:45:56.111518 22072 ts_tablet_manager.cc:1905] T 8ee6889a458743b9ab4bb495fa75dccc P adf8ab33231d4dd8ac6eabccb177dfb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:45:56.112649 22071 ts_tablet_manager.cc:1905] T 8ee6889a458743b9ab4bb495fa75dccc P 6981b6d0a4cf4606999908a3f8401924: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:45:56.116561 22073 ts_tablet_manager.cc:1905] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:45:56.118847 22072 ts_tablet_manager.cc:1918] T 8ee6889a458743b9ab4bb495fa75dccc P adf8ab33231d4dd8ac6eabccb177dfb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:45:56.119169 22072 log.cc:1196] T 8ee6889a458743b9ab4bb495fa75dccc P adf8ab33231d4dd8ac6eabccb177dfb5: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-1/wal/wals/8ee6889a458743b9ab4bb495fa75dccc
I20260926 16:45:56.120019 22071 ts_tablet_manager.cc:1918] T 8ee6889a458743b9ab4bb495fa75dccc P 6981b6d0a4cf4606999908a3f8401924: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:45:56.120088 22072 ts_tablet_manager.cc:1939] T 8ee6889a458743b9ab4bb495fa75dccc P adf8ab33231d4dd8ac6eabccb177dfb5: Deleting consensus metadata
I20260926 16:45:56.120318 22071 log.cc:1196] T 8ee6889a458743b9ab4bb495fa75dccc P 6981b6d0a4cf4606999908a3f8401924: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-2/wal/wals/8ee6889a458743b9ab4bb495fa75dccc
I20260926 16:45:56.121026 22071 ts_tablet_manager.cc:1939] T 8ee6889a458743b9ab4bb495fa75dccc P 6981b6d0a4cf4606999908a3f8401924: Deleting consensus metadata
I20260926 16:45:56.122345 21525 catalog_manager.cc:4841] TS adf8ab33231d4dd8ac6eabccb177dfb5 (127.9.238.194:42525): tablet 8ee6889a458743b9ab4bb495fa75dccc (table test-table [id=4a5b5e73f2744ce297420a5aed5d6ec8]) successfully deleted
I20260926 16:45:56.122674 21524 catalog_manager.cc:4841] TS 6981b6d0a4cf4606999908a3f8401924 (127.9.238.195:43041): tablet 8ee6889a458743b9ab4bb495fa75dccc (table test-table [id=4a5b5e73f2744ce297420a5aed5d6ec8]) successfully deleted
Sep 26 16:45:56 dist-test-slave-wt2j krb5kdc[21062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.193@KRBTEST.COM
I20260926 16:45:56.131011 22073 ts_tablet_manager.cc:1918] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:45:56.131305 22073 log.cc:1196] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790440988577176-10171-0/minicluster-data/ts-0/wal/wals/8ee6889a458743b9ab4bb495fa75dccc
I20260926 16:45:56.132010 22073 ts_tablet_manager.cc:1939] T 8ee6889a458743b9ab4bb495fa75dccc P ea4ae58d4e59494c9eb3bcf1ef7021e7: Deleting consensus metadata
I20260926 16:45:56.134539 21522 catalog_manager.cc:4841] TS ea4ae58d4e59494c9eb3bcf1ef7021e7 (127.9.238.193:36215): tablet 8ee6889a458743b9ab4bb495fa75dccc (table test-table [id=4a5b5e73f2744ce297420a5aed5d6ec8]) successfully deleted
I20260926 16:45:56.138616 22046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:56.104110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54856 (local address 127.9.238.193:36215)
0926 16:45:56.104429 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:56.104439 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:56.111022 (+ 6583us) server_negotiation.cc:407] Connection header received
0926 16:45:56.111707 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:56.111713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:56.111859 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:56.111952 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:45:56.117463 (+ 5511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:56.118650 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:56.121007 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:56.121770 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:56.121893 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:56.129506 (+ 7613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:45:56.129527 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:45:56.129553 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:45:56.129619 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:45:56.134607 (+ 4988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:56.135785 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:56.135803 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:56.135808 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:56.136066 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:56.136618 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:56.136624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:56.136627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:56.137153 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:45:56.137577 (+ 424us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:56.137917 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 16:45:56.138225 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260926 16:45:56.139755 21746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:45:56.244653 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 21618
I20260926 16:45:56.264231 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 21757
Sep 26 16:45:56 dist-test-slave-wt2j krb5kdc[21062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.194@KRBTEST.COM
I20260926 16:45:56.281663 22044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:56.265187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35136 (local address 127.9.238.194:42525)
0926 16:45:56.265482 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:56.265487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:56.265509 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:45:56.265607 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:56.265611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:56.265689 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:56.265785 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:45:56.267320 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:56.268330 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:56.269721 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:56.270127 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:56.270245 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:56.275304 (+ 5059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:45:56.275320 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:45:56.275344 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:45:56.275382 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:45:56.278722 (+ 3340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:56.279616 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:56.279623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:56.279626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:56.279764 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:56.280131 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:56.280134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:56.280136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:56.280490 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:45:56.280710 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:56.281087 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 16:45:56.281254 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260926 16:45:56.282460 21889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:45:56.376307 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 21757
I20260926 16:45:56.393520 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 21900
Sep 26 16:45:56 dist-test-slave-wt2j krb5kdc[21062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.195@KRBTEST.COM
I20260926 16:45:56.410275 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:45:56.394462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40198 (local address 127.9.238.195:43041)
0926 16:45:56.394638 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 16:45:56.394642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:45:56.394744 (+ 102us) server_negotiation.cc:407] Connection header received
0926 16:45:56.394933 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:45:56.394937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:45:56.395005 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:45:56.395112 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:45:56.396954 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:56.397682 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:56.399137 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:45:56.399513 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:45:56.399694 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:45:56.404544 (+ 4850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:45:56.404559 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:45:56.404577 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:45:56.404612 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:45:56.407360 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:56.408270 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:56.408278 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:56.408280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:56.408404 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:45:56.408821 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:45:56.408824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:45:56.408826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:45:56.409186 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:45:56.409403 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 16:45:56.409861 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 16:45:56.410021 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 16:45:56.411180 22023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:45:56.507807 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 21900
I20260926 16:45:56.525242 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 21506
I20260926 16:45:56.526619 21568 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:45:56.642997 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 21506
I20260926 16:45:56.672004 10171 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 16:45:57.019076 10171 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 16:45:57.019327 10171 mini_ranger.cc:66] Stopping Ranger...
I20260926 16:45:57.365638 10171 mini_ranger.cc:68] Stopped Ranger
2026-09-26 16:45:57.365 UTC [21092] LOG: received smart shutdown request
2026-09-26 16:45:57.370 UTC [21092] LOG: background worker "logical replication launcher" (PID 21101) exited with exit code 1
2026-09-26 16:45:57.370 UTC [21096] LOG: shutting down
2026-09-26 16:45:57.594 UTC [21092] LOG: database system is shut down
2026-09-26T16:45:57Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (57612 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:45:57 dist-test-slave-wt2j krb5kdc[22087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:45:57 dist-test-slave-wt2j krb5kdc[22087](info): set up 2 sockets
Sep 26 16:45:57 dist-test-slave-wt2j krb5kdc[22087](info): commencing operation
krb5kdc: starting...
W20260926 16:45:58.909241 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:45:59 dist-test-slave-wt2j krb5kdc[22087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:45:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:45:59Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.9.238.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.9.238.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.9.238.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/rangeradmin_127.9.238.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.9.238.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/rangeradmin_127.9.238.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.9.238.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.9.238.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.9.238.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/rangerlookup_127.9.238.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.9.238.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/rangerlookup_127.9.238.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.238.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.9.238.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.238.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/HTTP_127.9.238.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.238.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/HTTP_127.9.238.212@KRBTEST.COM.keytab.
I20260926 16:45:59.291157 10171 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskIzJvco/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/postgres -l logfile start
2026-09-26 16:46:02.221 UTC [22117] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 16:46:02.221 UTC [22117] LOG: listening on IPv4 address "127.9.238.212", port 55953
2026-09-26 16:46:02.230 UTC [22117] LOG: listening on Unix socket "/tmp/.s.PGSQL.55953"
2026-09-26 16:46:02.258 UTC [22120] LOG: database system was shut down at 2026-09-26 16:46:00 UTC
2026-09-26 16:46:02.262 UTC [22117] LOG: database system is ready to accept connections
I20260926 16:46:03.248785 10171 mini_postgres.cc:96] Postgres bound to 55953
2026-09-26 16:46:03.269 UTC [22128] FATAL: database "slave" does not exist
127.9.238.212:55953 - accepting connections
I20260926 16:46:03.270113 10171 mini_ranger.cc:161] Starting Ranger...
I20260926 16:46:03.301463 10171 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 16:46:03.943120 10171 mini_ranger.cc:87] Created ranger Postgres database
I20260926 16:46:03.943250 10171 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ranger-admin
2026-09-26 16:46:04,250 [I] DB FLAVOR :POSTGRES
2026-09-26 16:46:04,251 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 16:46:04,251 [I] Checking connection..
2026-09-26 16:46:04,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:55953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 16:46:04,526 [I] Checking connection passed.
2026-09-26 16:46:04,527 [I] --------- Verifying version history table ---------
2026-09-26 16:46:04,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:55953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 16:46:04,836 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 16:46:04,837 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:55953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 16:46:05,105 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 16:46:05,105 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 16:46:05,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:55953/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 16:46:05.366 UTC [22192] WARNING: there is no transaction in progress
2026-09-26 16:46:05,374 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 16:46:05,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:55953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 16:46:05,692 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 16:46:05,692 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 16:46:05,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:55953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 16:46:05,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:55953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 16:46:06,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:55953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-wt2j.c.gcp-upstream.internal','N') ;"
2026-09-26 16:46:06,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:55953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 16:46:06,905 [I] Table x_portal_user does not exist in database ranger
2026-09-26 16:46:06,905 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:55953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 16:46:07,202 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 16:46:07,202 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:55953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 16:46:07,462 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:55953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 16:46:07,731 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 16:46:07,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:55953/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 16:46:08.944 UTC [22300] WARNING: there is no transaction in progress
2026-09-26 16:46:08.964 UTC [22300] WARNING: there is no transaction in progress
2026-09-26 16:46:08.985 UTC [22300] WARNING: there is no transaction in progress
2026-09-26 16:46:09.006 UTC [22300] WARNING: there is no transaction in progress
2026-09-26 16:46:09.028 UTC [22300] WARNING: there is no transaction in progress
2026-09-26 16:46:09.053 UTC [22300] WARNING: there is no transaction in progress
2026-09-26 16:46:09.126 UTC [22300] WARNING: there is no transaction in progress
2026-09-26 16:46:09.134 UTC [22300] WARNING: there is no transaction in progress
2026-09-26 16:46:09.152 UTC [22300] WARNING: there is no transaction in progress
2026-09-26 16:46:09.165 UTC [22300] WARNING: there is no transaction in progress
2026-09-26 16:46:09.177 UTC [22300] WARNING: there is no transaction in progress
2026-09-26 16:46:09.201 UTC [22300] WARNING: there is no transaction in progress
2026-09-26 16:46:09.214 UTC [22300] WARNING: there is no transaction in progress
2026-09-26 16:46:09.224 UTC [22300] WARNING: there is no transaction in progress
2026-09-26 16:46:09.233 UTC [22300] WARNING: there is no transaction in progress
2026-09-26 16:46:09.242 UTC [22300] WARNING: there is no transaction in progress
2026-09-26 16:46:10,163 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 16:46:10,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:55953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 16:46:10,428 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 16:46:10,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:55953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 16:46:10,724 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 16:46:10,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:55953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 16:46:11,017 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 16:46:11,017 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:55953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 16:46:11,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:55953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 16:46:11,558 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:55953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-wt2j.c.gcp-upstream.internal';"
2026-09-26 16:46:11,838 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 16:46:11,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:55953/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 16:46:12,151 [I] DB_PATCHES have already been applied
I20260926 16:46:12.159107 10171 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ranger-admin:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskIzJvco/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 16:46:12.159317 10171 mini_ranger.cc:193] Using host: 127.9.238.212
I20260926 16:46:12.178002 10171 mini_ranger.cc:227] Ranger admin URL: http://127.9.238.212:56685
Sep 26, 2026 4:46:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 4:46:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 4:46:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 4:46:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 4:46:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 4:46:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 4:46:13 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 16:46:13.407720 10171 mini_ranger.cc:160] Time spent starting Ranger: real 10.138s user 0.007s sys 0.048s
Sep 26, 2026 4:46:35 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 4:46:35 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 4:46:35 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 4:46:35 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 4:46:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 4:46:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 4:46:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 4:46:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 4:46:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 4:46:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 4:46:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 4:46:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 4:46:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 4:46:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 4:46:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 4:46:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 4:46:35 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 4:46:36 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 16:46:37.544163 10171 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.9.238.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.9.238.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.9.238.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/rangerkms_127.9.238.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.9.238.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/rangerkms_127.9.238.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.9.238.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.9.238.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.9.238.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/HTTP_127.9.238.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.9.238.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/HTTP_127.9.238.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 16:46:37 dist-test-slave-wt2j krb5kdc[22087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441197, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 16:46:37.749581 10171 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 16:46:37.776161 10171 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 16:46:38.952313 10171 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 16:46:38.952477 10171 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ranger-kms
I20260926 16:46:38.952545 10171 mini_ranger_kms.cc:227] Using postgres at 127.9.238.212:55953
2026-09-26 16:46:39,386 [I] DB FLAVOR :POSTGRES
2026-09-26 16:46:39,387 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 16:46:39,387 [I] Checking connection
2026-09-26 16:46:39,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:55953/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 16:46:39,662 [I] connection success
2026-09-26 16:46:39,662 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 16:46:39,662 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 16:46:39,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:55953/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 16:46:39,922 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 16:46:39,922 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 16:46:39,922 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 16:46:39,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.9.238.212:55953/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 16:46:40,227 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 16:46:40.713338 10171 mini_ranger_kms.cc:310] Created kms service
I20260926 16:46:40.929052 10171 mini_ranger_kms.cc:326] Created kudu user
I20260926 16:46:40.986801 10171 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 16:46:40 dist-test-slave-wt2j krb5kdc[22087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441197, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.9.238.212@KRBTEST.COM
I20260926 16:46:41.491302 10171 mini_ranger_kms.cc:384] Added ranger policy
I20260926 16:46:41.491883 10171 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ranger-kms:/tmp/dist-test-taskIzJvco/build/asan/bin/postgresql.jar:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskIzJvco/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 16:46:41.491976 10171 mini_ranger_kms.cc:240] Using host: 127.9.238.212
I20260926 16:46:41.512365 10171 mini_ranger_kms.cc:276] Ranger KMS PID: 22481
I20260926 16:46:41.512538 10171 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.9.238.212:55571
Sep 26, 2026 4:46:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 4:46:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 4:46:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 4:46:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 4:46:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 4:46:43 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 16:46:43.915283 10171 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.166s user 0.045s sys 0.411s
I20260926 16:46:43.915639 10171 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 16:46:43.915875 10171 mini_ranger_kms.cc:401] 127.9.238.212:55571/kms/v1/keys
Sep 26 16:46:44 dist-test-slave-wt2j krb5kdc[22087](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790441204, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.238.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 16:46:44 dist-test-slave-wt2j krb5kdc[22087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441204, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.238.212@KRBTEST.COM for HTTP/127.9.238.212@KRBTEST.COM
Sep 26 16:46:45 dist-test-slave-wt2j krb5kdc[22087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441204, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.9.238.212@KRBTEST.COM for HTTP/127.9.238.212@KRBTEST.COM
Sep 26 16:46:47 dist-test-slave-wt2j krb5kdc[22087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:46:47.734364 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:43709
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:43709
--ranger_config_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.238.212:55571/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:48.170212 22531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:48.170622 22531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:48.170730 22531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:48.179195 22531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:48.179366 22531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:48.179417 22531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:48.179472 22531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:48.179527 22531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:48.192216 22531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/wal
--ranger_kms_url=127.9.238.212:55571/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:43709
--ranger_config_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:43709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:46:48.193943 22531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:48.195775 22531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:48.211401 22537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:48.211453 22536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:48.212116 22539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:48.212620 22531 server_base.cc:1034] running on GCE node
I20260926 16:46:48.213692 22531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:48.215777 22531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:48.217020 22531 hybrid_clock.cc:648] HybridClock initialized: now 1790441208216942 us; error 66 us; skew 500 ppm
I20260926 16:46:48.217490 22531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:46:48 dist-test-slave-wt2j krb5kdc[22087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441208, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:48.221678 22531 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:46:48.223778 22531 webserver.cc:460] Webserver started at http://127.9.238.254:44057/ using document root <none> and password file <none>
I20260926 16:46:48.224653 22531 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:48.224812 22531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:48.225195 22531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:46:48 dist-test-slave-wt2j krb5kdc[22087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441208, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for HTTP/127.9.238.212@KRBTEST.COM
I20260926 16:46:48.296213 22531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "b41f485addbb4a1598b2686505332bf4"
format_stamp: "Formatted at 2026-09-26 16:46:48 on dist-test-slave-wt2j"
server_key: "0ea41e6cf5bd4557124116ffb0b927a0"
server_key_iv: "34350db995a3f8f70ae165d70c0c8571"
server_key_version: "kuduclusterkey@0"
I20260926 16:46:48.297091 22531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "b41f485addbb4a1598b2686505332bf4"
format_stamp: "Formatted at 2026-09-26 16:46:48 on dist-test-slave-wt2j"
server_key: "0ea41e6cf5bd4557124116ffb0b927a0"
server_key_iv: "34350db995a3f8f70ae165d70c0c8571"
server_key_version: "kuduclusterkey@0"
I20260926 16:46:48.303679 22531 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 16:46:48.328573 22546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:48.330866 22531 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260926 16:46:48.331138 22531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "b41f485addbb4a1598b2686505332bf4"
format_stamp: "Formatted at 2026-09-26 16:46:48 on dist-test-slave-wt2j"
server_key: "0ea41e6cf5bd4557124116ffb0b927a0"
server_key_iv: "34350db995a3f8f70ae165d70c0c8571"
server_key_version: "kuduclusterkey@0"
I20260926 16:46:48.331439 22531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:48.354456 22531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:48.359289 22531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:48.359684 22531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:48.381997 22531 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:43709
I20260926 16:46:48.382061 22597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:43709 every 8 connection(s)
I20260926 16:46:48.383651 22531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:48.388999 22598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:48.390983 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 22531
I20260926 16:46:48.391249 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:46:48.392235 10171 external_mini_cluster.cc:1428] Setting key 248e3446df976f7d386b3cd59a930d8a
I20260926 16:46:48.409207 22598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4: Bootstrap starting.
Sep 26 16:46:48 dist-test-slave-wt2j krb5kdc[22087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:46:48.413931 22598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:48.415387 22598 log.cc:824] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:48.419865 22598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4: No bootstrap required, opened a new log
I20260926 16:46:48.421840 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:48.395092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60618 (local address 127.9.238.254:43709)
0926 16:46:48.396148 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:48.396165 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:48.396217 (+ 52us) server_negotiation.cc:407] Connection header received
0926 16:46:48.398075 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:48.398102 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:48.398658 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:48.399196 (+ 538us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:48.401519 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.403188 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.405496 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:48.406477 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:48.406692 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:48.412618 (+ 5926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:48.412628 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:48.412649 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:48.412704 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:48.416592 (+ 3888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.417798 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.417806 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.417814 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.417999 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:48.418535 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:48.418540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:48.418543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:48.419244 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:48.419494 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:48.420188 (+ 694us) server_negotiation.cc:299] Negotiation successful
0926 16:46:48.420665 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":515,"threads_started":1}
I20260926 16:46:48.429394 22598 raft_consensus.cc:348] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41f485addbb4a1598b2686505332bf4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 43709 } }
I20260926 16:46:48.429986 22598 raft_consensus.cc:374] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:48.430110 22598 raft_consensus.cc:729] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b41f485addbb4a1598b2686505332bf4, State: Initialized, Role: FOLLOWER
I20260926 16:46:48.430879 22598 consensus_queue.cc:260] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41f485addbb4a1598b2686505332bf4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 43709 } }
I20260926 16:46:48.431173 22598 raft_consensus.cc:388] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:46:48.431337 22598 raft_consensus.cc:482] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:46:48.431588 22598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:48.434535 22598 raft_consensus.cc:504] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41f485addbb4a1598b2686505332bf4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 43709 } }
I20260926 16:46:48.435387 22598 leader_election.cc:302] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b41f485addbb4a1598b2686505332bf4; no voters:
I20260926 16:46:48.436362 22598 leader_election.cc:288] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:46:48.436553 22603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:48.437745 22603 raft_consensus.cc:686] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [term 1 LEADER]: Becoming Leader. State: Replica: b41f485addbb4a1598b2686505332bf4, State: Running, Role: LEADER
I20260926 16:46:48.438472 22603 consensus_queue.cc:237] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41f485addbb4a1598b2686505332bf4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 43709 } }
I20260926 16:46:48.439379 22598 sys_catalog.cc:564] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:48.441756 22604 sys_catalog.cc:455] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b41f485addbb4a1598b2686505332bf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41f485addbb4a1598b2686505332bf4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 43709 } } }
I20260926 16:46:48.442281 22604 sys_catalog.cc:458] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:48.441885 22605 sys_catalog.cc:455] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b41f485addbb4a1598b2686505332bf4. Latest consensus state: current_term: 1 leader_uuid: "b41f485addbb4a1598b2686505332bf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41f485addbb4a1598b2686505332bf4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 43709 } } }
I20260926 16:46:48.443629 22605 sys_catalog.cc:458] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:48.448788 22609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:46:48.455502 22609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:46:48.456614 22598 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 16:46:48.476449 22609 catalog_manager.cc:1347] Generated new cluster ID: e38a8aabde7744e1bce543b678b87a34
I20260926 16:46:48.476653 22609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:46:48.498342 22609 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:46:48.501055 22609 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:46:48.511385 22609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4: Generated new TSK 0
I20260926 16:46:48.512881 22609 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
I20260926 16:46:49.317916 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.238.212:55571/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:0
--local_ip_for_outbound_sockets=127.9.238.193
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:43709
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:46:49 dist-test-slave-wt2j krb5kdc[22087](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790441209, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 16:46:49.792580 22643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:49.793068 22643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:49.793303 22643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:49.804666 22643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:49.804914 22643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:49.805159 22643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:46:49.821760 22643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.9.238.212:55571/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:43709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:46:49.823818 22643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:49.825578 22643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:49.847123 22653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:49.847055 22654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:49.854871 22643 server_base.cc:1034] running on GCE node
W20260926 16:46:49.856576 22656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:49.857752 22643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:49.859256 22643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:49.862740 22643 hybrid_clock.cc:648] HybridClock initialized: now 1790441209862644 us; error 101 us; skew 500 ppm
I20260926 16:46:49.863190 22643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:46:49 dist-test-slave-wt2j krb5kdc[22087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441209, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:49.875770 22643 init.cc:377] Logged in from keytab as kudu/127.9.238.193@KRBTEST.COM (short username kudu)
I20260926 16:46:49.880841 22643 webserver.cc:460] Webserver started at http://127.9.238.193:44265/ using document root <none> and password file <none>
I20260926 16:46:49.881737 22643 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:49.882009 22643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:49.882351 22643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:46:49 dist-test-slave-wt2j krb5kdc[22087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441209, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for HTTP/127.9.238.212@KRBTEST.COM
I20260926 16:46:49.908667 22643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/data/instance:
uuid: "4942bcf75c9e45aaae5829836936c673"
format_stamp: "Formatted at 2026-09-26 16:46:49 on dist-test-slave-wt2j"
server_key: "8249120a797e27122294be6ab6196fb8"
server_key_iv: "f003f3ddd488983b354d9582db1da772"
server_key_version: "kuduclusterkey@0"
I20260926 16:46:49.910022 22643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance:
uuid: "4942bcf75c9e45aaae5829836936c673"
format_stamp: "Formatted at 2026-09-26 16:46:49 on dist-test-slave-wt2j"
server_key: "8249120a797e27122294be6ab6196fb8"
server_key_iv: "f003f3ddd488983b354d9582db1da772"
server_key_version: "kuduclusterkey@0"
I20260926 16:46:49.923866 22643 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.004s sys 0.008s
I20260926 16:46:49.945688 22663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:49.948096 22643 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260926 16:46:49.948418 22643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "4942bcf75c9e45aaae5829836936c673"
format_stamp: "Formatted at 2026-09-26 16:46:49 on dist-test-slave-wt2j"
server_key: "8249120a797e27122294be6ab6196fb8"
server_key_iv: "f003f3ddd488983b354d9582db1da772"
server_key_version: "kuduclusterkey@0"
I20260926 16:46:49.948809 22643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:49.979257 22643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:49.986738 22643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:49.987286 22643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:49.995144 22643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:46:50.000422 22643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:46:50.000674 22643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260926 16:46:50.000964 22643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:46:50.001135 22643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:50.092705 22643 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:41709
I20260926 16:46:50.093084 22775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:41709 every 8 connection(s)
I20260926 16:46:50.095084 22643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:46:50.097652 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 22643
I20260926 16:46:50.097978 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance
I20260926 16:46:50.099310 10171 external_mini_cluster.cc:1428] Setting key a863382053540d3808be94409c334592
Sep 26 16:46:50 dist-test-slave-wt2j krb5kdc[22087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441209, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.238.194@KRBTEST.COM; defaulting to no policy
I20260926 16:46:50.142419 22779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:50.099911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:56489 (local address 127.9.238.254:43709)
0926 16:46:50.115167 (+ 15256us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:50.115174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:50.115195 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:46:50.115265 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:50.115270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:50.115375 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:50.115491 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:50.118649 (+ 3158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:50.119903 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:50.123780 (+ 3877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:50.124646 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:50.124801 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:50.130841 (+ 6040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:50.130851 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:50.130856 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:50.130927 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:50.136221 (+ 5294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:50.138217 (+ 1996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:50.138226 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:50.138231 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:50.138602 (+ 371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:50.139405 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:50.139412 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:50.139416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:50.139930 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:50.140178 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:50.141675 (+ 1497us) server_negotiation.cc:299] Negotiation successful
0926 16:46:50.141997 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15095,"thread_start_us":17107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.194@KRBTEST.COM" created.
I20260926 16:46:50.154150 22778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:50.099800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:43709 (local address 127.9.238.193:56489)
0926 16:46:50.111476 (+ 11676us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:50.111574 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:50.114258 (+ 2684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:50.116884 (+ 2626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:50.116897 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:50.117582 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:50.118418 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:50.118442 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:50.120134 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:50.120139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:50.122019 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:50.122034 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:50.125380 (+ 3346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:50.125393 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:50.125839 (+ 446us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:50.126968 (+ 1129us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:50.127011 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:50.130369 (+ 3358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:50.136705 (+ 6336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:50.136767 (+ 62us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:50.136797 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:50.137962 (+ 1165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:50.138820 (+ 858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:50.138828 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:50.138831 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:50.139223 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:50.140301 (+ 1078us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:50.140313 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:50.140984 (+ 671us) client_negotiation.cc:769] Sending connection context
0926 16:46:50.147774 (+ 6790us) client_negotiation.cc:240] Negotiation successful
0926 16:46:50.148787 (+ 1013us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11256,"thread_start_us":15727,"threads_started":1}
I20260926 16:46:50.157641 22776 heartbeater.cc:347] Connected to a master server at 127.9.238.254:43709
I20260926 16:46:50.158149 22776 heartbeater.cc:464] Registering TS with master...
I20260926 16:46:50.159225 22776 heartbeater.cc:511] Master 127.9.238.254:43709 requested a full tablet report, sending...
I20260926 16:46:50.162966 22563 ts_manager.cc:194] Registered new tserver with Master: 4942bcf75c9e45aaae5829836936c673 (127.9.238.193:41709)
I20260926 16:46:50.165668 22563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.193@KRBTEST.COM'} at 127.9.238.193:56489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
I20260926 16:46:50.184880 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.238.212:55571/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:0
--local_ip_for_outbound_sockets=127.9.238.194
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:43709
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:46:50 dist-test-slave-wt2j krb5kdc[22087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441209, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for HTTP/127.9.238.212@KRBTEST.COM
Sep 26 16:46:50 dist-test-slave-wt2j krb5kdc[22087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441209, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for HTTP/127.9.238.212@KRBTEST.COM
W20260926 16:46:50.782737 22782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:50.783306 22782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:50.783516 22782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:50.795446 22782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:50.796159 22782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:50.796437 22782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:46:50.813377 22782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.9.238.212:55571/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:43709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:46:50.815478 22782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:50.817337 22782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:50.847848 22789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:50.852643 22790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:50.868904 22793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:50.869168 22782 server_base.cc:1034] running on GCE node
I20260926 16:46:50.870399 22782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:50.880793 22782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:50.884409 22782 hybrid_clock.cc:648] HybridClock initialized: now 1790441210884259 us; error 141 us; skew 500 ppm
I20260926 16:46:50.885205 22782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:46:50 dist-test-slave-wt2j krb5kdc[22087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441210, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:50.891176 22782 init.cc:377] Logged in from keytab as kudu/127.9.238.194@KRBTEST.COM (short username kudu)
I20260926 16:46:50.901958 22782 webserver.cc:460] Webserver started at http://127.9.238.194:41033/ using document root <none> and password file <none>
I20260926 16:46:50.903348 22782 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:50.903592 22782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:50.904088 22782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:46:50 dist-test-slave-wt2j krb5kdc[22087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441210, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for HTTP/127.9.238.212@KRBTEST.COM
I20260926 16:46:50.929009 22782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/data/instance:
uuid: "9b3239c555974dc4801f38ad4237282e"
format_stamp: "Formatted at 2026-09-26 16:46:50 on dist-test-slave-wt2j"
server_key: "585c89037ab254ef8c76a74e0aa60df9"
server_key_iv: "87d5b4befc55bbf715b8495d1b402571"
server_key_version: "kuduclusterkey@0"
I20260926 16:46:50.932529 22782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance:
uuid: "9b3239c555974dc4801f38ad4237282e"
format_stamp: "Formatted at 2026-09-26 16:46:50 on dist-test-slave-wt2j"
server_key: "585c89037ab254ef8c76a74e0aa60df9"
server_key_iv: "87d5b4befc55bbf715b8495d1b402571"
server_key_version: "kuduclusterkey@0"
I20260926 16:46:50.971972 22782 fs_manager.cc:691] Time spent creating directory manager: real 0.039s user 0.021s sys 0.016s
I20260926 16:46:50.994155 22806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:50.995960 22782 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 16:46:50.996172 22782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "9b3239c555974dc4801f38ad4237282e"
format_stamp: "Formatted at 2026-09-26 16:46:50 on dist-test-slave-wt2j"
server_key: "585c89037ab254ef8c76a74e0aa60df9"
server_key_iv: "87d5b4befc55bbf715b8495d1b402571"
server_key_version: "kuduclusterkey@0"
I20260926 16:46:50.996429 22782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:51.018010 22782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:51.023064 22782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:51.023432 22782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:51.024593 22782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:46:51.026173 22782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:46:51.026319 22782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 16:46:51.026490 22782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:46:51.026567 22782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:51.077001 22782 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:33873
I20260926 16:46:51.077004 22918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:33873 every 8 connection(s)
I20260926 16:46:51.078567 22782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:46:51.087996 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 22782
I20260926 16:46:51.088231 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance
I20260926 16:46:51.089048 10171 external_mini_cluster.cc:1428] Setting key 7276a32950987ec5a65c8d64208c27d3
Sep 26 16:46:51 dist-test-slave-wt2j krb5kdc[22087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441210, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:46:51.163409 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:51.083347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:43709 (local address 127.9.238.194:48027)
0926 16:46:51.120498 (+ 37151us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:51.120598 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:51.126866 (+ 6268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:51.141362 (+ 14496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:51.141372 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:51.141929 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:51.142575 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:51.142606 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:51.144371 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:51.144377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:51.145823 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:51.145838 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:51.147066 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:51.147073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:51.147335 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:51.148335 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:51.148372 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:51.151847 (+ 3475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:51.157483 (+ 5636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:51.157498 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:51.157525 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:51.158304 (+ 779us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:51.158849 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:51.158858 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:51.158865 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:51.159196 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:51.160065 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:51.160077 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:51.160628 (+ 551us) client_negotiation.cc:769] Sending connection context
0926 16:46:51.160963 (+ 335us) client_negotiation.cc:240] Negotiation successful
0926 16:46:51.161665 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36628,"thread_start_us":39441,"threads_started":1}
I20260926 16:46:51.165359 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:51.083953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:48027 (local address 127.9.238.254:43709)
0926 16:46:51.125434 (+ 41481us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:51.125445 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:51.125486 (+ 41us) server_negotiation.cc:407] Connection header received
0926 16:46:51.140575 (+ 15089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:51.140583 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:51.140701 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:51.140779 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:51.142786 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:51.144210 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:51.146000 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:51.146797 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:51.146961 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:51.152202 (+ 5241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:51.152213 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:51.152222 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:51.152282 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:51.157216 (+ 4934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:51.158514 (+ 1298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:51.158523 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:51.158530 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:51.158702 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:51.159380 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:51.159389 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:51.159396 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:51.159894 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:51.164376 (+ 4482us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:51.164555 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 16:46:51.164860 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41237,"thread_start_us":41526,"threads_started":1}
I20260926 16:46:51.166946 22919 heartbeater.cc:347] Connected to a master server at 127.9.238.254:43709
I20260926 16:46:51.167322 22919 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.9.238.195@KRBTEST.COM; defaulting to no policy
I20260926 16:46:51.168270 22919 heartbeater.cc:511] Master 127.9.238.254:43709 requested a full tablet report, sending...
I20260926 16:46:51.170054 22563 ts_manager.cc:194] Registered new tserver with Master: 9b3239c555974dc4801f38ad4237282e (127.9.238.194:33873)
I20260926 16:46:51.171069 22563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.194@KRBTEST.COM'} at 127.9.238.194:48027
I20260926 16:46:51.171276 22776 heartbeater.cc:503] Master 127.9.238.254:43709 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
I20260926 16:46:51.204341 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.238.212:55571/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:0
--local_ip_for_outbound_sockets=127.9.238.195
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:43709
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:51.575699 22925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:51.576027 22925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:51.576162 22925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:51.583968 22925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:51.584100 22925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:51.584283 22925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:46:51.594867 22925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.9.238.212:55571/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:43709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:46:51.596366 22925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:51.597695 22925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:51.614626 22931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:51.614532 22930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:51.615324 22933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:51.618062 22925 server_base.cc:1034] running on GCE node
I20260926 16:46:51.619292 22925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:51.620832 22925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:51.622170 22925 hybrid_clock.cc:648] HybridClock initialized: now 1790441211622081 us; error 82 us; skew 500 ppm
I20260926 16:46:51.622674 22925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:46:51 dist-test-slave-wt2j krb5kdc[22087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441211, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:51.626909 22925 init.cc:377] Logged in from keytab as kudu/127.9.238.195@KRBTEST.COM (short username kudu)
I20260926 16:46:51.629065 22925 webserver.cc:460] Webserver started at http://127.9.238.195:34569/ using document root <none> and password file <none>
I20260926 16:46:51.630057 22925 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:51.630229 22925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:51.630652 22925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 16:46:51 dist-test-slave-wt2j krb5kdc[22087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441211, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for HTTP/127.9.238.212@KRBTEST.COM
I20260926 16:46:51.651254 22925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/data/instance:
uuid: "e82a7304e9f3406495ee33f7612e7692"
format_stamp: "Formatted at 2026-09-26 16:46:51 on dist-test-slave-wt2j"
server_key: "492108a67ab2534c51063fa1b487530d"
server_key_iv: "50c27b8d623f4b737b37ca128dbd2d01"
server_key_version: "kuduclusterkey@0"
I20260926 16:46:51.652510 22925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance:
uuid: "e82a7304e9f3406495ee33f7612e7692"
format_stamp: "Formatted at 2026-09-26 16:46:51 on dist-test-slave-wt2j"
server_key: "492108a67ab2534c51063fa1b487530d"
server_key_iv: "50c27b8d623f4b737b37ca128dbd2d01"
server_key_version: "kuduclusterkey@0"
I20260926 16:46:51.660115 22925 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260926 16:46:51.682118 22940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:51.684434 22925 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:46:51.684648 22925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "e82a7304e9f3406495ee33f7612e7692"
format_stamp: "Formatted at 2026-09-26 16:46:51 on dist-test-slave-wt2j"
server_key: "492108a67ab2534c51063fa1b487530d"
server_key_iv: "50c27b8d623f4b737b37ca128dbd2d01"
server_key_version: "kuduclusterkey@0"
I20260926 16:46:51.684872 22925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:51.709306 22925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:51.714370 22925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:51.714669 22925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:51.715857 22925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:46:51.717491 22925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:46:51.717579 22925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:51.717676 22925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:46:51.717729 22925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:51.759692 22925 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:43853
I20260926 16:46:51.759871 23052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:43853 every 8 connection(s)
I20260926 16:46:51.761144 22925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:46:51.762532 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 22925
I20260926 16:46:51.762778 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance
I20260926 16:46:51.763437 10171 external_mini_cluster.cc:1428] Setting key 630b228c509879667b2c158b9ead7927
Sep 26 16:46:51 dist-test-slave-wt2j krb5kdc[22087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441211, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:46:51.787247 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:51.765014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:35833 (local address 127.9.238.254:43709)
0926 16:46:51.765641 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:51.765646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:51.766942 (+ 1296us) server_negotiation.cc:407] Connection header received
0926 16:46:51.769341 (+ 2399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:51.769346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:51.769477 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:51.769609 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:51.771459 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:51.772336 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:51.774217 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:51.774733 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:51.774894 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:51.779649 (+ 4755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:51.779657 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:51.779661 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:51.779740 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:51.783382 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:51.784479 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:51.784485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:51.784488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:51.784652 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:51.785265 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:51.785269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:51.785271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:51.785627 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:51.785784 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:51.786708 (+ 924us) server_negotiation.cc:299] Negotiation successful
0926 16:46:51.786947 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":316,"threads_started":1}
I20260926 16:46:51.788337 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:51.765630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:43709 (local address 127.9.238.195:35833)
0926 16:46:51.766760 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:51.766817 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:51.769007 (+ 2190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:51.769920 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:51.769930 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:51.770464 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:51.771250 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:51.771273 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:51.772519 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:51.772524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:51.774055 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:51.774068 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:51.774998 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:51.775005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:51.775305 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:51.776221 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:51.776406 (+ 185us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:51.779264 (+ 2858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:51.783675 (+ 4411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:51.783685 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:51.783701 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:51.784302 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:51.784800 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:51.784804 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:51.784806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:51.785125 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:51.785788 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:51.785794 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:51.786305 (+ 511us) client_negotiation.cc:769] Sending connection context
0926 16:46:51.786576 (+ 271us) client_negotiation.cc:240] Negotiation successful
0926 16:46:51.787084 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":419,"threads_started":1}
I20260926 16:46:51.790583 23053 heartbeater.cc:347] Connected to a master server at 127.9.238.254:43709
I20260926 16:46:51.791029 23053 heartbeater.cc:464] Registering TS with master...
I20260926 16:46:51.792089 23053 heartbeater.cc:511] Master 127.9.238.254:43709 requested a full tablet report, sending...
I20260926 16:46:51.793917 22563 ts_manager.cc:194] Registered new tserver with Master: e82a7304e9f3406495ee33f7612e7692 (127.9.238.195:43853)
I20260926 16:46:51.795048 22563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.195@KRBTEST.COM'} at 127.9.238.195:35833
I20260926 16:46:51.795229 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:46:51.796986 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 22643
Sep 26 16:46:51 dist-test-slave-wt2j krb5kdc[22087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.193@KRBTEST.COM
I20260926 16:46:51.818857 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:51.798235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43726 (local address 127.9.238.193:41709)
0926 16:46:51.798886 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:51.798893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:51.798924 (+ 31us) server_negotiation.cc:407] Connection header received
0926 16:46:51.799012 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:51.799018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:51.799232 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:51.799383 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:51.801924 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:51.803023 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:51.805269 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:51.805893 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:51.806085 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:51.811939 (+ 5854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:51.811955 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:51.811972 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:51.812035 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:51.815060 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:51.816188 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:51.816200 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:51.816203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:51.816530 (+ 327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:51.817035 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:51.817040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:51.817042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:51.817565 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:51.817880 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:51.818174 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 16:46:51.818524 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":304,"threads_started":1}
I20260926 16:46:51.820681 22771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:46:51.916531 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 22643
I20260926 16:46:51.935882 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 22782
Sep 26 16:46:51 dist-test-slave-wt2j krb5kdc[22087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.194@KRBTEST.COM
I20260926 16:46:51.961968 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:51.937027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48964 (local address 127.9.238.194:33873)
0926 16:46:51.937808 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:51.937815 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:51.937845 (+ 30us) server_negotiation.cc:407] Connection header received
0926 16:46:51.937940 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:51.937947 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:51.938214 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:51.938372 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:51.940255 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:51.941625 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:51.943254 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:51.943915 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:51.944082 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:51.954803 (+ 10721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:51.954824 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:51.954842 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:51.954893 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:51.958426 (+ 3533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:51.959663 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:51.959673 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:51.959675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:51.959859 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:51.960305 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:51.960309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:51.960311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:51.960738 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:51.961020 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:51.961364 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 16:46:51.961631 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":461,"threads_started":1}
I20260926 16:46:51.964061 22914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:46:52.061559 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 22782
I20260926 16:46:52.080014 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 22925
Sep 26 16:46:52 dist-test-slave-wt2j krb5kdc[22087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.195@KRBTEST.COM
I20260926 16:46:52.101706 23062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:52.081510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38128 (local address 127.9.238.195:43853)
0926 16:46:52.082332 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:52.082340 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:52.082376 (+ 36us) server_negotiation.cc:407] Connection header received
0926 16:46:52.082447 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:52.082454 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:52.082713 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:52.082867 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:52.085010 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:52.086079 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:52.087638 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:52.088237 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:52.088387 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:52.093740 (+ 5353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:52.093763 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:52.093785 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:52.093844 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:52.097798 (+ 3954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:52.098945 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:52.098955 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:52.098958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:52.099150 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:52.099744 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:52.099768 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:52.099771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:52.100247 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:52.100598 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:52.100985 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 16:46:52.101296 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":396,"threads_started":1}
I20260926 16:46:52.103629 23048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:46:52.197676 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 22925
I20260926 16:46:52.214519 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 22531
I20260926 16:46:52.216080 22593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:46:52.328430 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 22531
I20260926 16:46:52.357887 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:43709
--webserver_interface=127.9.238.254
--webserver_port=44057
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:43709
--ranger_config_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.238.212:55571/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.238.212:55571/kms with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:52.790237 23065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:52.790704 23065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:52.790905 23065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:52.799975 23065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:52.800184 23065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:52.800303 23065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:52.800410 23065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:52.800513 23065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:52.812849 23065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.238.212:55571/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:43709
--ranger_config_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:43709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=44057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:46:52.814903 23065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:52.816728 23065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:52.832562 23070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:52.832353 23073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:52.832504 23071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:52.834055 23065 server_base.cc:1034] running on GCE node
I20260926 16:46:52.835144 23065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:52.837275 23065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:52.838570 23065 hybrid_clock.cc:648] HybridClock initialized: now 1790441212838501 us; error 65 us; skew 500 ppm
I20260926 16:46:52.839075 23065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:46:52 dist-test-slave-wt2j krb5kdc[22087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441212, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:52.843197 23065 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:46:52.845356 23065 webserver.cc:460] Webserver started at http://127.9.238.254:44057/ using document root <none> and password file <none>
I20260926 16:46:52.846213 23065 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:52.846340 23065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 16:46:52 dist-test-slave-wt2j krb5kdc[22087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441212, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for HTTP/127.9.238.212@KRBTEST.COM
I20260926 16:46:52.922961 23065 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 16:46:52.927865 23081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:52.930018 23065 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260926 16:46:52.930274 23065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "b41f485addbb4a1598b2686505332bf4"
format_stamp: "Formatted at 2026-09-26 16:46:48 on dist-test-slave-wt2j"
server_key: "0ea41e6cf5bd4557124116ffb0b927a0"
server_key_iv: "34350db995a3f8f70ae165d70c0c8571"
server_key_version: "kuduclusterkey@0"
I20260926 16:46:52.931659 23065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:52.960229 23065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:52.965667 23065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:52.965996 23065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:52.992411 23065 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:43709
I20260926 16:46:52.992437 23132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:43709 every 8 connection(s)
I20260926 16:46:52.994446 23065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:53.000571 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 23065
I20260926 16:46:53.001780 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.238.212:55571/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:41709
--local_ip_for_outbound_sockets=127.9.238.193
--tserver_master_addrs=127.9.238.254:43709
--webserver_port=44265
--webserver_interface=127.9.238.193
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.238.212:55571/kms with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:46:53.002810 23133 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:46:53.006449 23133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4: Bootstrap starting.
I20260926 16:46:53.027230 23133 log.cc:824] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:53.036911 23133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:46:53.037549 23133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4: Bootstrap complete.
I20260926 16:46:53.047293 23133 raft_consensus.cc:348] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41f485addbb4a1598b2686505332bf4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 43709 } }
I20260926 16:46:53.048425 23133 raft_consensus.cc:729] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b41f485addbb4a1598b2686505332bf4, State: Initialized, Role: FOLLOWER
I20260926 16:46:53.049144 23133 consensus_queue.cc:260] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41f485addbb4a1598b2686505332bf4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 43709 } }
I20260926 16:46:53.049400 23133 raft_consensus.cc:388] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:46:53.049531 23133 raft_consensus.cc:482] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:46:53.049734 23133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [term 1 FOLLOWER]: Advancing to term 2
I20260926 16:46:53.054872 23133 raft_consensus.cc:504] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41f485addbb4a1598b2686505332bf4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 43709 } }
I20260926 16:46:53.055662 23133 leader_election.cc:302] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b41f485addbb4a1598b2686505332bf4; no voters:
I20260926 16:46:53.056468 23133 leader_election.cc:288] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 16:46:53.056638 23137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 16:46:53.058176 23137 raft_consensus.cc:686] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [term 2 LEADER]: Becoming Leader. State: Replica: b41f485addbb4a1598b2686505332bf4, State: Running, Role: LEADER
I20260926 16:46:53.058982 23137 consensus_queue.cc:237] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41f485addbb4a1598b2686505332bf4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 43709 } }
I20260926 16:46:53.059384 23133 sys_catalog.cc:564] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:46:53.062129 23139 sys_catalog.cc:455] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b41f485addbb4a1598b2686505332bf4. Latest consensus state: current_term: 2 leader_uuid: "b41f485addbb4a1598b2686505332bf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41f485addbb4a1598b2686505332bf4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 43709 } } }
I20260926 16:46:53.062880 23139 sys_catalog.cc:458] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:53.063848 23138 sys_catalog.cc:455] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b41f485addbb4a1598b2686505332bf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41f485addbb4a1598b2686505332bf4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 43709 } } }
I20260926 16:46:53.064242 23138 sys_catalog.cc:458] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:46:53.068248 23143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:46:53.078977 23143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:46:53.080658 23143 catalog_manager.cc:1259] Loaded cluster ID: e38a8aabde7744e1bce543b678b87a34
I20260926 16:46:53.080816 23143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:46:53.085119 23143 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:46:53.085350 23133 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 16:46:53.086969 23143 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4: Loaded TSK: 0
I20260926 16:46:53.088758 23143 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 16:46:53.506712 23135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:53.507267 23135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:53.507490 23135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:53.519807 23135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:53.520074 23135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:53.520306 23135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:46:53.536825 23135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.238.212:55571/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:41709
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=44265
--tserver_master_addrs=127.9.238.254:43709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:46:53.539098 23135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:53.541060 23135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:53.562999 23170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:53.564982 23171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:53.571489 23135 server_base.cc:1034] running on GCE node
W20260926 16:46:53.576050 23174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:53.577627 23135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:53.587395 23135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:53.591059 23135 hybrid_clock.cc:648] HybridClock initialized: now 1790441213590921 us; error 150 us; skew 500 ppm
I20260926 16:46:53.591943 23135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:46:53 dist-test-slave-wt2j krb5kdc[22087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441213, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:53.598205 23135 init.cc:377] Logged in from keytab as kudu/127.9.238.193@KRBTEST.COM (short username kudu)
I20260926 16:46:53.602293 23135 webserver.cc:460] Webserver started at http://127.9.238.193:44265/ using document root <none> and password file <none>
I20260926 16:46:53.603780 23135 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:53.604012 23135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 16:46:53 dist-test-slave-wt2j krb5kdc[22087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441213, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for HTTP/127.9.238.212@KRBTEST.COM
I20260926 16:46:53.671380 23135 fs_manager.cc:709] Time spent opening directory manager: real 0.027s user 0.003s sys 0.009s
I20260926 16:46:53.675889 23181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:53.681703 23135 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.001s
I20260926 16:46:53.681991 23135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "4942bcf75c9e45aaae5829836936c673"
format_stamp: "Formatted at 2026-09-26 16:46:49 on dist-test-slave-wt2j"
server_key: "8249120a797e27122294be6ab6196fb8"
server_key_iv: "f003f3ddd488983b354d9582db1da772"
server_key_version: "kuduclusterkey@0"
I20260926 16:46:53.683696 23135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:53.705042 23135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:53.710988 23135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:53.711354 23135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:53.712881 23135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:46:53.729923 23135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:46:53.730209 23135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 16:46:53.730427 23135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:46:53.730577 23135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:53.798050 23135 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:41709
I20260926 16:46:53.798156 23294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:41709 every 8 connection(s)
I20260926 16:46:53.799916 23135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:46:53.806145 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 23135
I20260926 16:46:53.807094 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.238.212:55571/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:33873
--local_ip_for_outbound_sockets=127.9.238.194
--tserver_master_addrs=127.9.238.254:43709
--webserver_port=41033
--webserver_interface=127.9.238.194
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.238.212:55571/kms with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:46:53 dist-test-slave-wt2j krb5kdc[22087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441213, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:46:53.836318 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:53.803492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:43709 (local address 127.9.238.193:43921)
0926 16:46:53.804947 (+ 1455us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:53.804998 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:53.807672 (+ 2674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:53.810379 (+ 2707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:53.810392 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:53.811090 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:53.812008 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:53.812038 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:53.814603 (+ 2565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:53.814611 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:53.816788 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:53.816804 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:53.818463 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:53.818474 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:53.818811 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:53.820065 (+ 1254us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:53.820116 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:53.823859 (+ 3743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:53.830195 (+ 6336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:53.830209 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:53.830238 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:53.830865 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:53.831441 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:53.831447 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:53.831452 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:53.831792 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:53.833000 (+ 1208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:53.833011 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:53.833534 (+ 523us) client_negotiation.cc:769] Sending connection context
0926 16:46:53.833837 (+ 303us) client_negotiation.cc:240] Negotiation successful
0926 16:46:53.834414 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1153,"thread_start_us":852,"threads_started":1}
I20260926 16:46:53.840217 23297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:53.803621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:43921 (local address 127.9.238.254:43709)
0926 16:46:53.804597 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:53.804626 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:53.805632 (+ 1006us) server_negotiation.cc:407] Connection header received
0926 16:46:53.808625 (+ 2993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:53.808645 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:53.809197 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:53.809601 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:53.812332 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:53.814379 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:53.816965 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:53.818150 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:53.819477 (+ 1327us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:53.824443 (+ 4966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:53.824457 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:53.824480 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:53.824544 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:53.829927 (+ 5383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:53.831081 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:53.831091 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:53.831097 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:53.831294 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:53.831934 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:53.831940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:53.831951 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:53.832809 (+ 858us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:53.837727 (+ 4918us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:53.838063 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 16:46:53.838542 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":352,"threads_started":1}
I20260926 16:46:53.844619 23295 heartbeater.cc:347] Connected to a master server at 127.9.238.254:43709
I20260926 16:46:53.845122 23295 heartbeater.cc:464] Registering TS with master...
I20260926 16:46:53.846381 23295 heartbeater.cc:511] Master 127.9.238.254:43709 requested a full tablet report, sending...
W20260926 16:46:53.851967 23295 heartbeater.cc:644] Failed to heartbeat to 127.9.238.254:43709 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 16:46:53.852521 23295 heartbeater.cc:464] Registering TS with master...
I20260926 16:46:53.853063 23295 heartbeater.cc:511] Master 127.9.238.254:43709 requested a full tablet report, sending...
I20260926 16:46:53.855046 23295 heartbeater.cc:464] Registering TS with master...
I20260926 16:46:53.855573 23295 heartbeater.cc:511] Master 127.9.238.254:43709 requested a full tablet report, sending...
W20260926 16:46:53.857285 23295 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 16:46:54.242637 23299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:54.243091 23299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:54.243271 23299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:54.252296 23299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:54.252440 23299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:54.252635 23299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:46:54.265220 23299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.238.212:55571/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:33873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=41033
--tserver_master_addrs=127.9.238.254:43709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:46:54.267169 23299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:54.269023 23299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:54.288996 23312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:54.289143 23314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:54.289541 23311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:54.297804 23299 server_base.cc:1034] running on GCE node
I20260926 16:46:54.299031 23299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:54.300752 23299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:54.302846 23299 hybrid_clock.cc:648] HybridClock initialized: now 1790441214302774 us; error 52 us; skew 500 ppm
I20260926 16:46:54.303339 23299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:46:54 dist-test-slave-wt2j krb5kdc[22087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441214, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:54.308373 23299 init.cc:377] Logged in from keytab as kudu/127.9.238.194@KRBTEST.COM (short username kudu)
I20260926 16:46:54.318259 23299 webserver.cc:460] Webserver started at http://127.9.238.194:41033/ using document root <none> and password file <none>
I20260926 16:46:54.319343 23299 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:54.319613 23299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 16:46:54 dist-test-slave-wt2j krb5kdc[22087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441214, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for HTTP/127.9.238.212@KRBTEST.COM
I20260926 16:46:54.354454 23299 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:46:54.359006 23324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:54.361073 23299 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260926 16:46:54.361320 23299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "9b3239c555974dc4801f38ad4237282e"
format_stamp: "Formatted at 2026-09-26 16:46:50 on dist-test-slave-wt2j"
server_key: "585c89037ab254ef8c76a74e0aa60df9"
server_key_iv: "87d5b4befc55bbf715b8495d1b402571"
server_key_version: "kuduclusterkey@0"
I20260926 16:46:54.362622 23299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:54.392092 23299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:54.398677 23299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:54.399091 23299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:54.409901 23299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:46:54.413331 23299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:46:54.413486 23299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:54.413630 23299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:46:54.413715 23299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:54.474656 23299 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:33873
I20260926 16:46:54.474754 23438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:33873 every 8 connection(s)
I20260926 16:46:54.476991 23299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:46:54.483913 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 23299
I20260926 16:46:54.484766 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.9.238.212:55571/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:43853
--local_ip_for_outbound_sockets=127.9.238.195
--tserver_master_addrs=127.9.238.254:43709
--webserver_port=34569
--webserver_interface=127.9.238.195
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:35673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.9.238.212:55571/kms with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:46:54 dist-test-slave-wt2j krb5kdc[22087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441214, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:46:54.514946 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:54.480725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:43003 (local address 127.9.238.254:43709)
0926 16:46:54.484509 (+ 3784us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:54.484517 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:54.485241 (+ 724us) server_negotiation.cc:407] Connection header received
0926 16:46:54.488591 (+ 3350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:54.488598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:54.488725 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:54.488837 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:54.491187 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:54.492924 (+ 1737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:54.496004 (+ 3080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:54.496832 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:54.496973 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:54.502038 (+ 5065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:54.502049 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:54.502058 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:54.502110 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:54.507918 (+ 5808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:54.510540 (+ 2622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:54.510549 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:54.510556 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:54.510760 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:54.511433 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:54.511441 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:54.511447 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:54.511949 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:54.512124 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:54.514101 (+ 1977us) server_negotiation.cc:299] Negotiation successful
0926 16:46:54.514411 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3607,"thread_start_us":3493,"threads_started":1}
I20260926 16:46:54.517863 23442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:54.481245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:43709 (local address 127.9.238.194:43003)
0926 16:46:54.484834 (+ 3589us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:54.485068 (+ 234us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:54.487999 (+ 2931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:54.489272 (+ 1273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:54.489288 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:54.490036 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:54.490948 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:54.490976 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:54.493120 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:54.493126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:54.495816 (+ 2690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:54.495836 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:54.497307 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:54.497316 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:54.497653 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:54.498682 (+ 1029us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:54.498726 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:54.501706 (+ 2980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:54.508564 (+ 6858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:54.508580 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:54.508607 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:54.510325 (+ 1718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:54.510925 (+ 600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:54.510933 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:54.510938 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:54.511267 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:54.512208 (+ 941us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:54.512219 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:54.512832 (+ 613us) client_negotiation.cc:769] Sending connection context
0926 16:46:54.515664 (+ 2832us) client_negotiation.cc:240] Negotiation successful
0926 16:46:54.516382 (+ 718us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2678,"thread_start_us":7032,"threads_started":1}
I20260926 16:46:54.521173 23439 heartbeater.cc:347] Connected to a master server at 127.9.238.254:43709
I20260926 16:46:54.521504 23439 heartbeater.cc:464] Registering TS with master...
I20260926 16:46:54.522393 23439 heartbeater.cc:511] Master 127.9.238.254:43709 requested a full tablet report, sending...
I20260926 16:46:54.524518 23098 ts_manager.cc:194] Registered new tserver with Master: 9b3239c555974dc4801f38ad4237282e (127.9.238.194:33873)
Sep 26 16:46:54 dist-test-slave-wt2j krb5kdc[22087](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790441214, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:54.526511 23098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.194@KRBTEST.COM'} at 127.9.238.194:43003
I20260926 16:46:54.857445 23295 heartbeater.cc:464] Registering TS with master...
I20260926 16:46:54.857954 23295 heartbeater.cc:511] Master 127.9.238.254:43709 requested a full tablet report, sending...
I20260926 16:46:54.859383 23098 ts_manager.cc:194] Registered new tserver with Master: 4942bcf75c9e45aaae5829836936c673 (127.9.238.193:41709)
I20260926 16:46:54.860677 23098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.193@KRBTEST.COM'} at 127.9.238.193:43921
Sep 26 16:46:54 dist-test-slave-wt2j krb5kdc[22087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441214, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for HTTP/127.9.238.212@KRBTEST.COM
W20260926 16:46:55.093844 23443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:55.094236 23443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:55.094403 23443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:55.103332 23443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:55.103579 23443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:55.103813 23443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:46:55.116515 23443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:35673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.9.238.212:55571/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:43853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=34569
--tserver_master_addrs=127.9.238.254:43709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:46:55.118353 23443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:55.120246 23443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:55.142340 23449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:55.145123 23450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:55.150002 23452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:55.151786 23443 server_base.cc:1034] running on GCE node
I20260926 16:46:55.152803 23443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:55.154312 23443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:55.155701 23443 hybrid_clock.cc:648] HybridClock initialized: now 1790441215155609 us; error 78 us; skew 500 ppm
I20260926 16:46:55.156186 23443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:46:55 dist-test-slave-wt2j krb5kdc[22087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441215, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:55.160422 23443 init.cc:377] Logged in from keytab as kudu/127.9.238.195@KRBTEST.COM (short username kudu)
I20260926 16:46:55.162949 23443 webserver.cc:460] Webserver started at http://127.9.238.195:34569/ using document root <none> and password file <none>
I20260926 16:46:55.163990 23443 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:55.164176 23443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 16:46:55 dist-test-slave-wt2j krb5kdc[22087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441215, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for HTTP/127.9.238.212@KRBTEST.COM
I20260926 16:46:55.195843 23443 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.005s sys 0.004s
I20260926 16:46:55.199674 23460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:55.201741 23443 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:46:55.201979 23443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "e82a7304e9f3406495ee33f7612e7692"
format_stamp: "Formatted at 2026-09-26 16:46:51 on dist-test-slave-wt2j"
server_key: "492108a67ab2534c51063fa1b487530d"
server_key_iv: "50c27b8d623f4b737b37ca128dbd2d01"
server_key_version: "kuduclusterkey@0"
I20260926 16:46:55.203223 23443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:55.223865 23443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:55.230728 23443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:55.231271 23443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:55.236178 23443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:46:55.239144 23443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:46:55.239351 23443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260926 16:46:55.239537 23443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:46:55.239714 23443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:55.323629 23443 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:43853
I20260926 16:46:55.323659 23573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:43853 every 8 connection(s)
I20260926 16:46:55.326256 23443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:46:55.334034 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 23443
Sep 26 16:46:55 dist-test-slave-wt2j krb5kdc[22087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441214, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for HTTP/127.9.238.212@KRBTEST.COM
I20260926 16:46:55.380563 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:55.335941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45318 (local address 127.9.238.254:43709)
0926 16:46:55.350286 (+ 14345us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:55.350291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:55.350309 (+ 18us) server_negotiation.cc:407] Connection header received
0926 16:46:55.350378 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:55.350383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:55.350475 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:55.350585 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:55.358416 (+ 7831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.359604 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.361833 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.362603 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.362732 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:55.372552 (+ 9820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:55.372560 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:55.372564 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:55.372614 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:55.376507 (+ 3893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:55.377400 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:55.377405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:55.377408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:55.377566 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:55.377996 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:55.378000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:55.378003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:55.378325 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:55.378594 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:55.379903 (+ 1309us) server_negotiation.cc:299] Negotiation successful
0926 16:46:55.380169 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14243,"thread_start_us":12949,"threads_started":1}
Sep 26 16:46:55 dist-test-slave-wt2j krb5kdc[22087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441215, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:46:55.398514 23579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:55.333776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:32839 (local address 127.9.238.254:43709)
0926 16:46:55.349155 (+ 15379us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:55.349162 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:55.357064 (+ 7902us) server_negotiation.cc:407] Connection header received
0926 16:46:55.357138 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:55.357143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:55.357239 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:55.357335 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:55.368695 (+ 11360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.369925 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.381241 (+ 11316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.382107 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.382244 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:55.388828 (+ 6584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:55.388838 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:55.388843 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:55.388896 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:55.392932 (+ 4036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:55.394213 (+ 1281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:55.394220 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:55.394223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:55.394427 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:55.395133 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:55.395138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:55.395141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:55.395652 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:55.395813 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:55.397604 (+ 1791us) server_negotiation.cc:299] Negotiation successful
0926 16:46:55.397954 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15231,"thread_start_us":16259,"threads_started":1}
I20260926 16:46:55.402534 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:55.332480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:43709 (local address 127.9.238.195:32839)
0926 16:46:55.354066 (+ 21586us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:55.354133 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:55.356579 (+ 2446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:55.364200 (+ 7621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:55.364214 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:55.364926 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:55.365830 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:55.365856 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.370255 (+ 4399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.370261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:55.372268 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:55.372293 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.383661 (+ 11368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.383673 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:55.384161 (+ 488us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:55.385237 (+ 1076us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:55.385283 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:55.388380 (+ 3097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:55.393188 (+ 4808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:55.393201 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:55.393231 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:55.394031 (+ 800us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:55.394615 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:55.394620 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:55.394623 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:55.394983 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:55.395923 (+ 940us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:55.395936 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:55.397011 (+ 1075us) client_negotiation.cc:769] Sending connection context
0926 16:46:55.399710 (+ 2699us) client_negotiation.cc:240] Negotiation successful
0926 16:46:55.400586 (+ 876us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21070,"thread_start_us":24811,"threads_started":1}
I20260926 16:46:55.405224 23574 heartbeater.cc:347] Connected to a master server at 127.9.238.254:43709
I20260926 16:46:55.405679 23574 heartbeater.cc:464] Registering TS with master...
I20260926 16:46:55.406879 23574 heartbeater.cc:511] Master 127.9.238.254:43709 requested a full tablet report, sending...
I20260926 16:46:55.409508 23098 ts_manager.cc:194] Registered new tserver with Master: e82a7304e9f3406495ee33f7612e7692 (127.9.238.195:43853)
I20260926 16:46:55.410928 23098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.195@KRBTEST.COM'} at 127.9.238.195:32839
I20260926 16:46:55.412132 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:46:55.470841 23579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:55.419411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45320 (local address 127.9.238.254:43709)
0926 16:46:55.419686 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:55.419693 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:55.432384 (+ 12691us) server_negotiation.cc:407] Connection header received
0926 16:46:55.432708 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:55.432715 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:55.432808 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:55.432889 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:55.435122 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.436338 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.438754 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.439441 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.439576 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:55.455374 (+ 15798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:55.455384 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:55.455389 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:55.455460 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:55.459283 (+ 3823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:55.467788 (+ 8505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:55.467797 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:55.467802 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:55.468059 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:55.468669 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:55.468674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:55.468678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:55.469088 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:55.469382 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:55.470035 (+ 653us) server_negotiation.cc:299] Negotiation successful
0926 16:46:55.470454 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 16:46:55.478569 23097 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45320:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:46:55.491791 23097 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:46:55.531527 23439 heartbeater.cc:503] Master 127.9.238.254:43709 was elected leader, sending a full tablet report...
I20260926 16:46:55.554103 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:55.513481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48978 (local address 127.9.238.194:33873)
0926 16:46:55.540979 (+ 27498us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:55.540992 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:55.541031 (+ 39us) server_negotiation.cc:407] Connection header received
0926 16:46:55.541155 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:55.541167 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:55.541532 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:55.541757 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:55.542492 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.546148 (+ 3656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.549845 (+ 3697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.551451 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.551608 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:55.551871 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:55.553546 (+ 1675us) server_negotiation.cc:299] Negotiation successful
0926 16:46:55.553743 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27267,"thread_start_us":28475,"threads_started":1}
I20260926 16:46:55.562466 23589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:55.512797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:33873 (local address 127.0.0.1:48978)
0926 16:46:55.513488 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:55.513529 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:55.513791 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:55.541769 (+ 27978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:55.541777 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:55.541810 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:55.542260 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:55.542276 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.546373 (+ 4097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.546382 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:55.549582 (+ 3200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:55.549608 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.552067 (+ 2459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.552073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:55.552291 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:55.552355 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 16:46:55.552501 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 16:46:55.552731 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":902,"threads_started":1}
I20260926 16:46:55.568051 23592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:55.516087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:41709 (local address 127.0.0.1:43742)
0926 16:46:55.538399 (+ 22312us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:55.538422 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:55.538585 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:55.551584 (+ 12999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:55.551589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:55.551622 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:55.552000 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:55.552012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.554407 (+ 2395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.554412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:55.559467 (+ 5055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:55.559493 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.565683 (+ 6190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.565691 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:55.567414 (+ 1723us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:55.567454 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 16:46:55.567526 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 16:46:55.567702 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22140,"threads_started":1}
I20260926 16:46:55.569521 23593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:55.516731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43742 (local address 127.9.238.193:41709)
0926 16:46:55.550591 (+ 33860us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:55.550602 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:55.550636 (+ 34us) server_negotiation.cc:407] Connection header received
0926 16:46:55.550759 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:55.550767 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:55.551072 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:55.552226 (+ 1154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:55.552489 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.554184 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.559771 (+ 5587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.562101 (+ 2330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.568592 (+ 6491us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:55.568880 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:55.568989 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 16:46:55.569188 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33643,"thread_start_us":34504,"threads_started":1}
I20260926 16:46:55.569022 23374 tablet_service.cc:1467] Processing CreateTablet for tablet 4e82fa008a3a4531ad5cc68ad7742542 (DEFAULT_TABLE table=test-table [id=1019d22cfa1946d5841ff92292b24217]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:46:55.571126 23374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e82fa008a3a4531ad5cc68ad7742542. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:55.573662 23594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:55.516988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38138 (local address 127.9.238.195:43853)
0926 16:46:55.542682 (+ 25694us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:55.542693 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:55.565141 (+ 22448us) server_negotiation.cc:407] Connection header received
0926 16:46:55.565373 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:55.565383 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:55.565703 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:55.565923 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:55.566672 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.568333 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.570655 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.572224 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.572361 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:55.572596 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:55.573131 (+ 535us) server_negotiation.cc:299] Negotiation successful
0926 16:46:55.573305 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25477,"thread_start_us":30570,"threads_started":1}
I20260926 16:46:55.574378 23589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:55.516497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:43853 (local address 127.0.0.1:38138)
0926 16:46:55.564981 (+ 48484us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:55.565015 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:55.565176 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:55.566265 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:55.566270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:55.566298 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:55.566481 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:55.566491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.568568 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.568573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:55.570387 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:55.570400 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.572713 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.572720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:55.572893 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:55.572926 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 16:46:55.573899 (+ 973us) client_negotiation.cc:240] Negotiation successful
0926 16:46:55.574068 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48301,"thread_start_us":22194,"threads_started":1}
I20260926 16:46:55.575762 23230 tablet_service.cc:1467] Processing CreateTablet for tablet 4e82fa008a3a4531ad5cc68ad7742542 (DEFAULT_TABLE table=test-table [id=1019d22cfa1946d5841ff92292b24217]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:46:55.578117 23230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e82fa008a3a4531ad5cc68ad7742542. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:55.578929 23508 tablet_service.cc:1467] Processing CreateTablet for tablet 4e82fa008a3a4531ad5cc68ad7742542 (DEFAULT_TABLE table=test-table [id=1019d22cfa1946d5841ff92292b24217]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:46:55.580724 23508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e82fa008a3a4531ad5cc68ad7742542. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:55.603627 23596 tablet_bootstrap.cc:492] T 4e82fa008a3a4531ad5cc68ad7742542 P 4942bcf75c9e45aaae5829836936c673: Bootstrap starting.
I20260926 16:46:55.611675 23595 tablet_bootstrap.cc:492] T 4e82fa008a3a4531ad5cc68ad7742542 P e82a7304e9f3406495ee33f7612e7692: Bootstrap starting.
I20260926 16:46:55.613569 23597 tablet_bootstrap.cc:492] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e: Bootstrap starting.
I20260926 16:46:55.616928 23595 tablet_bootstrap.cc:654] T 4e82fa008a3a4531ad5cc68ad7742542 P e82a7304e9f3406495ee33f7612e7692: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:55.617679 23597 tablet_bootstrap.cc:654] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:55.618615 23595 log.cc:824] T 4e82fa008a3a4531ad5cc68ad7742542 P e82a7304e9f3406495ee33f7612e7692: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:55.619215 23597 log.cc:824] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:55.621011 23596 tablet_bootstrap.cc:654] T 4e82fa008a3a4531ad5cc68ad7742542 P 4942bcf75c9e45aaae5829836936c673: Neither blocks nor log segments found. Creating new log.
I20260926 16:46:55.622732 23596 log.cc:824] T 4e82fa008a3a4531ad5cc68ad7742542 P 4942bcf75c9e45aaae5829836936c673: Log is configured to *not* fsync() on all Append() calls
I20260926 16:46:55.627311 23595 tablet_bootstrap.cc:492] T 4e82fa008a3a4531ad5cc68ad7742542 P e82a7304e9f3406495ee33f7612e7692: No bootstrap required, opened a new log
I20260926 16:46:55.627849 23595 ts_tablet_manager.cc:1397] T 4e82fa008a3a4531ad5cc68ad7742542 P e82a7304e9f3406495ee33f7612e7692: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.004s
I20260926 16:46:55.627533 23597 tablet_bootstrap.cc:492] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e: No bootstrap required, opened a new log
I20260926 16:46:55.628512 23597 ts_tablet_manager.cc:1397] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.003s
I20260926 16:46:55.639891 23596 tablet_bootstrap.cc:492] T 4e82fa008a3a4531ad5cc68ad7742542 P 4942bcf75c9e45aaae5829836936c673: No bootstrap required, opened a new log
I20260926 16:46:55.640321 23596 ts_tablet_manager.cc:1397] T 4e82fa008a3a4531ad5cc68ad7742542 P 4942bcf75c9e45aaae5829836936c673: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.003s
I20260926 16:46:55.641675 23597 raft_consensus.cc:348] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3239c555974dc4801f38ad4237282e" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33873 } } peers { permanent_uuid: "4942bcf75c9e45aaae5829836936c673" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41709 } } peers { permanent_uuid: "e82a7304e9f3406495ee33f7612e7692" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 43853 } }
I20260926 16:46:55.642262 23597 raft_consensus.cc:374] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:55.642367 23597 raft_consensus.cc:729] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b3239c555974dc4801f38ad4237282e, State: Initialized, Role: FOLLOWER
I20260926 16:46:55.643298 23597 consensus_queue.cc:260] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3239c555974dc4801f38ad4237282e" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33873 } } peers { permanent_uuid: "4942bcf75c9e45aaae5829836936c673" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41709 } } peers { permanent_uuid: "e82a7304e9f3406495ee33f7612e7692" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 43853 } }
I20260926 16:46:55.653108 23597 ts_tablet_manager.cc:1428] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e: Time spent starting tablet: real 0.024s user 0.016s sys 0.004s
I20260926 16:46:55.654255 23595 raft_consensus.cc:348] T 4e82fa008a3a4531ad5cc68ad7742542 P e82a7304e9f3406495ee33f7612e7692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3239c555974dc4801f38ad4237282e" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33873 } } peers { permanent_uuid: "4942bcf75c9e45aaae5829836936c673" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41709 } } peers { permanent_uuid: "e82a7304e9f3406495ee33f7612e7692" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 43853 } }
I20260926 16:46:55.654887 23595 raft_consensus.cc:374] T 4e82fa008a3a4531ad5cc68ad7742542 P e82a7304e9f3406495ee33f7612e7692 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:55.655040 23595 raft_consensus.cc:729] T 4e82fa008a3a4531ad5cc68ad7742542 P e82a7304e9f3406495ee33f7612e7692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e82a7304e9f3406495ee33f7612e7692, State: Initialized, Role: FOLLOWER
I20260926 16:46:55.655097 23596 raft_consensus.cc:348] T 4e82fa008a3a4531ad5cc68ad7742542 P 4942bcf75c9e45aaae5829836936c673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3239c555974dc4801f38ad4237282e" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33873 } } peers { permanent_uuid: "4942bcf75c9e45aaae5829836936c673" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41709 } } peers { permanent_uuid: "e82a7304e9f3406495ee33f7612e7692" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 43853 } }
I20260926 16:46:55.655737 23596 raft_consensus.cc:374] T 4e82fa008a3a4531ad5cc68ad7742542 P 4942bcf75c9e45aaae5829836936c673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:46:55.655905 23596 raft_consensus.cc:729] T 4e82fa008a3a4531ad5cc68ad7742542 P 4942bcf75c9e45aaae5829836936c673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4942bcf75c9e45aaae5829836936c673, State: Initialized, Role: FOLLOWER
I20260926 16:46:55.655786 23595 consensus_queue.cc:260] T 4e82fa008a3a4531ad5cc68ad7742542 P e82a7304e9f3406495ee33f7612e7692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3239c555974dc4801f38ad4237282e" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33873 } } peers { permanent_uuid: "4942bcf75c9e45aaae5829836936c673" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41709 } } peers { permanent_uuid: "e82a7304e9f3406495ee33f7612e7692" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 43853 } }
I20260926 16:46:55.656706 23596 consensus_queue.cc:260] T 4e82fa008a3a4531ad5cc68ad7742542 P 4942bcf75c9e45aaae5829836936c673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3239c555974dc4801f38ad4237282e" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33873 } } peers { permanent_uuid: "4942bcf75c9e45aaae5829836936c673" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41709 } } peers { permanent_uuid: "e82a7304e9f3406495ee33f7612e7692" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 43853 } }
I20260926 16:46:55.666623 23596 ts_tablet_manager.cc:1428] T 4e82fa008a3a4531ad5cc68ad7742542 P 4942bcf75c9e45aaae5829836936c673: Time spent starting tablet: real 0.026s user 0.012s sys 0.011s
I20260926 16:46:55.666903 23601 raft_consensus.cc:482] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:46:55.667331 23295 heartbeater.cc:503] Master 127.9.238.254:43709 was elected leader, sending a full tablet report...
I20260926 16:46:55.667940 23601 raft_consensus.cc:504] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3239c555974dc4801f38ad4237282e" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33873 } } peers { permanent_uuid: "4942bcf75c9e45aaae5829836936c673" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41709 } } peers { permanent_uuid: "e82a7304e9f3406495ee33f7612e7692" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 43853 } }
I20260926 16:46:55.680469 23601 leader_election.cc:288] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4942bcf75c9e45aaae5829836936c673 (127.9.238.193:41709), e82a7304e9f3406495ee33f7612e7692 (127.9.238.195:43853)
I20260926 16:46:55.686489 23593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:55.672362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:41919 (local address 127.9.238.193:41709)
0926 16:46:55.672665 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:55.672672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:55.677350 (+ 4678us) server_negotiation.cc:407] Connection header received
0926 16:46:55.677687 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:55.677693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:55.677796 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:55.677865 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:55.678705 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.679807 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.682878 (+ 3071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.684738 (+ 1860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.684935 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:55.685069 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:55.685941 (+ 872us) server_negotiation.cc:299] Negotiation successful
0926 16:46:55.686122 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 16:46:55.687788 23605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:55.671872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:41709 (local address 127.9.238.194:41919)
0926 16:46:55.677172 (+ 5300us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:55.677212 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:55.677500 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:55.678099 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:55.678103 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:55.678125 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:55.678529 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:55.678543 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.679980 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.679985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:55.682625 (+ 2640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:55.682650 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.685369 (+ 2719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.685380 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:55.685682 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:55.685738 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 16:46:55.687083 (+ 1345us) client_negotiation.cc:240] Negotiation successful
0926 16:46:55.687292 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5116,"thread_start_us":4917,"threads_started":1}
I20260926 16:46:55.688987 23574 heartbeater.cc:503] Master 127.9.238.254:43709 was elected leader, sending a full tablet report...
I20260926 16:46:55.689815 23595 ts_tablet_manager.cc:1428] T 4e82fa008a3a4531ad5cc68ad7742542 P e82a7304e9f3406495ee33f7612e7692: Time spent starting tablet: real 0.062s user 0.018s sys 0.025s
I20260926 16:46:55.691511 23250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e82fa008a3a4531ad5cc68ad7742542" candidate_uuid: "9b3239c555974dc4801f38ad4237282e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4942bcf75c9e45aaae5829836936c673" is_pre_election: true
I20260926 16:46:55.692219 23250 raft_consensus.cc:2454] T 4e82fa008a3a4531ad5cc68ad7742542 P 4942bcf75c9e45aaae5829836936c673 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b3239c555974dc4801f38ad4237282e in term 0.
I20260926 16:46:55.693425 23325 leader_election.cc:302] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4942bcf75c9e45aaae5829836936c673, 9b3239c555974dc4801f38ad4237282e; no voters:
I20260926 16:46:55.693910 23601 raft_consensus.cc:2781] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:46:55.694082 23601 raft_consensus.cc:482] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:46:55.694193 23601 raft_consensus.cc:3037] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:55.698585 23601 raft_consensus.cc:504] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3239c555974dc4801f38ad4237282e" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33873 } } peers { permanent_uuid: "4942bcf75c9e45aaae5829836936c673" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41709 } } peers { permanent_uuid: "e82a7304e9f3406495ee33f7612e7692" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 43853 } }
I20260926 16:46:55.700117 23250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e82fa008a3a4531ad5cc68ad7742542" candidate_uuid: "9b3239c555974dc4801f38ad4237282e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4942bcf75c9e45aaae5829836936c673"
I20260926 16:46:55.700404 23250 raft_consensus.cc:3037] T 4e82fa008a3a4531ad5cc68ad7742542 P 4942bcf75c9e45aaae5829836936c673 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:55.701261 23594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:55.681366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:42723 (local address 127.9.238.195:43853)
0926 16:46:55.681651 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:55.681658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:55.686353 (+ 4695us) server_negotiation.cc:407] Connection header received
0926 16:46:55.686936 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:55.686942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:55.687029 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:55.687095 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:46:55.688705 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.689905 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.698257 (+ 8352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.699668 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.699808 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:55.699890 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:55.700778 (+ 888us) server_negotiation.cc:299] Negotiation successful
0926 16:46:55.700958 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":79}
I20260926 16:46:55.701992 23606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:55.681586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:43853 (local address 127.9.238.194:42723)
0926 16:46:55.686223 (+ 4637us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:55.686247 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:55.686705 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:55.687302 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:55.687306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:55.687334 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:46:55.687777 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:55.687791 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.690098 (+ 2307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.690104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:55.698022 (+ 7918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:55.698045 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.700373 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.700381 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:55.700565 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:55.700603 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 16:46:55.701449 (+ 846us) client_negotiation.cc:240] Negotiation successful
0926 16:46:55.701639 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4491,"thread_start_us":4791,"threads_started":1}
I20260926 16:46:55.704349 23601 leader_election.cc:288] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e [CANDIDATE]: Term 1 election: Requested vote from peers 4942bcf75c9e45aaae5829836936c673 (127.9.238.193:41709), e82a7304e9f3406495ee33f7612e7692 (127.9.238.195:43853)
I20260926 16:46:55.703481 23250 raft_consensus.cc:2454] T 4e82fa008a3a4531ad5cc68ad7742542 P 4942bcf75c9e45aaae5829836936c673 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b3239c555974dc4801f38ad4237282e in term 1.
I20260926 16:46:55.705107 23325 leader_election.cc:302] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4942bcf75c9e45aaae5829836936c673, 9b3239c555974dc4801f38ad4237282e; no voters:
I20260926 16:46:55.705127 23527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e82fa008a3a4531ad5cc68ad7742542" candidate_uuid: "9b3239c555974dc4801f38ad4237282e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e82a7304e9f3406495ee33f7612e7692"
I20260926 16:46:55.705540 23601 raft_consensus.cc:2781] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:46:55.705615 23527 raft_consensus.cc:3037] T 4e82fa008a3a4531ad5cc68ad7742542 P e82a7304e9f3406495ee33f7612e7692 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:46:55.705765 23601 raft_consensus.cc:686] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e [term 1 LEADER]: Becoming Leader. State: Replica: 9b3239c555974dc4801f38ad4237282e, State: Running, Role: LEADER
I20260926 16:46:55.706187 23528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e82fa008a3a4531ad5cc68ad7742542" candidate_uuid: "9b3239c555974dc4801f38ad4237282e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e82a7304e9f3406495ee33f7612e7692" is_pre_election: true
I20260926 16:46:55.706435 23601 consensus_queue.cc:237] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3239c555974dc4801f38ad4237282e" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33873 } } peers { permanent_uuid: "4942bcf75c9e45aaae5829836936c673" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41709 } } peers { permanent_uuid: "e82a7304e9f3406495ee33f7612e7692" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 43853 } }
I20260926 16:46:55.710604 23527 raft_consensus.cc:2454] T 4e82fa008a3a4531ad5cc68ad7742542 P e82a7304e9f3406495ee33f7612e7692 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b3239c555974dc4801f38ad4237282e in term 1.
I20260926 16:46:55.712813 23097 catalog_manager.cc:5495] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b3239c555974dc4801f38ad4237282e (127.9.238.194). New cstate: current_term: 1 leader_uuid: "9b3239c555974dc4801f38ad4237282e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b3239c555974dc4801f38ad4237282e" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 33873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4942bcf75c9e45aaae5829836936c673" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e82a7304e9f3406495ee33f7612e7692" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 43853 } health_report { overall_health: UNKNOWN } } }
W20260926 16:46:55.731598 23440 tablet.cc:2369] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:46:55.771824 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:55.764935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48992 (local address 127.9.238.194:33873)
0926 16:46:55.765211 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:55.765216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:55.765241 (+ 25us) server_negotiation.cc:407] Connection header received
0926 16:46:55.765427 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:55.765431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:55.765554 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:55.765725 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:46:55.766179 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.767090 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.768896 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.769489 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.769615 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:55.770313 (+ 698us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:46:55.771096 (+ 783us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:46:55.771229 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:55.771376 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 16:46:55.771471 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 16:46:55.782025 23527 raft_consensus.cc:1261] T 4e82fa008a3a4531ad5cc68ad7742542 P e82a7304e9f3406495ee33f7612e7692 [term 1 FOLLOWER]: Refusing update from remote peer 9b3239c555974dc4801f38ad4237282e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:46:55.782122 23250 raft_consensus.cc:1261] T 4e82fa008a3a4531ad5cc68ad7742542 P 4942bcf75c9e45aaae5829836936c673 [term 1 FOLLOWER]: Refusing update from remote peer 9b3239c555974dc4801f38ad4237282e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:46:55.783356 23604 consensus_queue.cc:1035] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4942bcf75c9e45aaae5829836936c673" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:46:55.783810 23601 consensus_queue.cc:1035] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e82a7304e9f3406495ee33f7612e7692" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 43853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:46:55.798988 23616 mvcc.cc:205] Tried to move back new op lower bound from 7333647219829272576 to 7333647219538722816. Current Snapshot: MvccSnapshot[applied={T|T < 7333647219829272576}]
W20260926 16:46:55.803515 23296 tablet.cc:2369] T 4e82fa008a3a4531ad5cc68ad7742542 P 4942bcf75c9e45aaae5829836936c673: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:46:55.807838 23618 mvcc.cc:205] Tried to move back new op lower bound from 7333647219829272576 to 7333647219538722816. Current Snapshot: MvccSnapshot[applied={T|T < 7333647219829272576}]
I20260926 16:46:55.826994 23098 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:46:55.827371 23098 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45320:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:46:55.832420 23098 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b41f485addbb4a1598b2686505332bf4: Sending DeleteTablet for 3 replicas of tablet 4e82fa008a3a4531ad5cc68ad7742542
I20260926 16:46:55.833906 23230 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e82fa008a3a4531ad5cc68ad7742542 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:46:55 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:43742
I20260926 16:46:55.834136 23508 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e82fa008a3a4531ad5cc68ad7742542 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:46:55 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:38138
I20260926 16:46:55.835027 23374 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e82fa008a3a4531ad5cc68ad7742542 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:46:55 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:48978
I20260926 16:46:55.835215 23627 tablet_replica.cc:330] T 4e82fa008a3a4531ad5cc68ad7742542 P e82a7304e9f3406495ee33f7612e7692: stopping tablet replica
I20260926 16:46:55.835856 23627 raft_consensus.cc:2229] T 4e82fa008a3a4531ad5cc68ad7742542 P e82a7304e9f3406495ee33f7612e7692 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:46:55.836791 23627 raft_consensus.cc:2258] T 4e82fa008a3a4531ad5cc68ad7742542 P e82a7304e9f3406495ee33f7612e7692 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:46:55.842065 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 23135
I20260926 16:46:55.847354 23626 tablet_replica.cc:330] T 4e82fa008a3a4531ad5cc68ad7742542 P 4942bcf75c9e45aaae5829836936c673: stopping tablet replica
I20260926 16:46:55.847718 23628 tablet_replica.cc:330] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e: stopping tablet replica
I20260926 16:46:55.847946 23626 raft_consensus.cc:2229] T 4e82fa008a3a4531ad5cc68ad7742542 P 4942bcf75c9e45aaae5829836936c673 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:46:55.848305 23628 raft_consensus.cc:2229] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:46:55.846453 23627 ts_tablet_manager.cc:1905] T 4e82fa008a3a4531ad5cc68ad7742542 P e82a7304e9f3406495ee33f7612e7692: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:46:55.849088 23628 raft_consensus.cc:2258] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:46:55.849833 23626 raft_consensus.cc:2258] T 4e82fa008a3a4531ad5cc68ad7742542 P 4942bcf75c9e45aaae5829836936c673 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:46:55.852392 23626 ts_tablet_manager.cc:1905] T 4e82fa008a3a4531ad5cc68ad7742542 P 4942bcf75c9e45aaae5829836936c673: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:46:55.855190 23628 ts_tablet_manager.cc:1905] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:46:55.860517 23626 ts_tablet_manager.cc:1918] T 4e82fa008a3a4531ad5cc68ad7742542 P 4942bcf75c9e45aaae5829836936c673: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:46:55.860797 23626 log.cc:1196] T 4e82fa008a3a4531ad5cc68ad7742542 P 4942bcf75c9e45aaae5829836936c673: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-0/wal/wals/4e82fa008a3a4531ad5cc68ad7742542
I20260926 16:46:55.861496 23626 ts_tablet_manager.cc:1939] T 4e82fa008a3a4531ad5cc68ad7742542 P 4942bcf75c9e45aaae5829836936c673: Deleting consensus metadata
I20260926 16:46:55.863121 23593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:55.843524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43750 (local address 127.9.238.193:41709)
0926 16:46:55.843778 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:55.843785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:55.843811 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:46:55.843902 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:55.843911 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:55.844027 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:55.844168 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:55.846648 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.848114 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.849968 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.850748 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.850938 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:55.854289 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:55.854307 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:55.854329 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:55.854385 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:55.859151 (+ 4766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:55.860205 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:55.860216 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:55.860220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:55.860522 (+ 302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:55.860998 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:55.861003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:55.861006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:55.861618 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:55.862052 (+ 434us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:55.862306 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 16:46:55.862633 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260926 16:46:55.863729 23082 catalog_manager.cc:4841] TS 4942bcf75c9e45aaae5829836936c673 (127.9.238.193:41709): tablet 4e82fa008a3a4531ad5cc68ad7742542 (table test-table [id=1019d22cfa1946d5841ff92292b24217]) successfully deleted
I20260926 16:46:55.864462 23290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:46:55.866338 23628 ts_tablet_manager.cc:1918] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:46:55.866653 23628 log.cc:1196] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-1/wal/wals/4e82fa008a3a4531ad5cc68ad7742542
I20260926 16:46:55.867362 23628 ts_tablet_manager.cc:1939] T 4e82fa008a3a4531ad5cc68ad7742542 P 9b3239c555974dc4801f38ad4237282e: Deleting consensus metadata
I20260926 16:46:55.867329 23627 ts_tablet_manager.cc:1918] T 4e82fa008a3a4531ad5cc68ad7742542 P e82a7304e9f3406495ee33f7612e7692: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:46:55.867501 23627 log.cc:1196] T 4e82fa008a3a4531ad5cc68ad7742542 P e82a7304e9f3406495ee33f7612e7692: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790440988577176-10171-0/minicluster-data/ts-2/wal/wals/4e82fa008a3a4531ad5cc68ad7742542
I20260926 16:46:55.868134 23627 ts_tablet_manager.cc:1939] T 4e82fa008a3a4531ad5cc68ad7742542 P e82a7304e9f3406495ee33f7612e7692: Deleting consensus metadata
I20260926 16:46:55.869635 23085 catalog_manager.cc:4841] TS 9b3239c555974dc4801f38ad4237282e (127.9.238.194:33873): tablet 4e82fa008a3a4531ad5cc68ad7742542 (table test-table [id=1019d22cfa1946d5841ff92292b24217]) successfully deleted
I20260926 16:46:55.869935 23085 catalog_manager.cc:4841] TS e82a7304e9f3406495ee33f7612e7692 (127.9.238.195:43853): tablet 4e82fa008a3a4531ad5cc68ad7742542 (table test-table [id=1019d22cfa1946d5841ff92292b24217]) successfully deleted
I20260926 16:46:55.963732 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 23135
I20260926 16:46:55.990296 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 23299
I20260926 16:46:56.017450 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:55.991319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49002 (local address 127.9.238.194:33873)
0926 16:46:55.991527 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:55.991532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:55.991553 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:46:55.991817 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:55.991821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:55.991962 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:55.992166 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:55.993418 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.994294 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.995670 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:55.996145 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:55.996284 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:56.001247 (+ 4963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:56.001269 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:56.001291 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:56.001344 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:56.012998 (+ 11654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:56.014475 (+ 1477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:56.014488 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:56.014492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:56.014753 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:56.015262 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:56.015268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:56.015272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:56.015859 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:56.016208 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:56.016617 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 16:46:56.016925 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 16:46:56.018312 23434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:46:56.152100 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 23299
I20260926 16:46:56.173821 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 23443
I20260926 16:46:56.190039 23594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:56.174778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38152 (local address 127.9.238.195:43853)
0926 16:46:56.175031 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:56.175039 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:56.175116 (+ 77us) server_negotiation.cc:407] Connection header received
0926 16:46:56.175378 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:56.175384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:56.175484 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:56.175662 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:56.177156 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:56.177990 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:56.179417 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:56.179949 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:56.180096 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:56.183040 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:56.183058 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:56.183082 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:56.183123 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:56.186817 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:56.187652 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:56.187662 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:56.187665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:56.187836 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:56.188262 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:56.188268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:56.188272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:56.188702 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:56.189045 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:56.189382 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 16:46:56.189631 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260926 16:46:56.191093 23569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:46:56.287132 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 23443
I20260926 16:46:56.307108 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 23065
I20260926 16:46:56.308682 23128 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:46:56.426501 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 23065
I20260926 16:46:56.453977 10171 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 16:46:56.800853 10171 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 16:46:56.801200 10171 mini_ranger.cc:66] Stopping Ranger...
I20260926 16:46:57.148063 10171 mini_ranger.cc:68] Stopped Ranger
2026-09-26 16:46:57.148 UTC [22117] LOG: received smart shutdown request
2026-09-26 16:46:57.152 UTC [22117] LOG: background worker "logical replication launcher" (PID 22126) exited with exit code 1
2026-09-26 16:46:57.153 UTC [22121] LOG: shutting down
2026-09-26 16:46:57.370 UTC [22117] LOG: database system is shut down
2026-09-26T16:46:57Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (59768 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:46:57 dist-test-slave-wt2j krb5kdc[23641](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:46:57 dist-test-slave-wt2j krb5kdc[23641](info): set up 2 sockets
Sep 26 16:46:57 dist-test-slave-wt2j krb5kdc[23641](info): commencing operation
krb5kdc: starting...
W20260926 16:46:58.685755 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:46:58 dist-test-slave-wt2j krb5kdc[23641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:46:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:46:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:46:58.968364 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:40205
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:40205,127.9.238.253:43311,127.9.238.252:45131
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:46:59.332373 23654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:59.332755 23654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:59.332930 23654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:59.340639 23654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:59.340811 23654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:59.340927 23654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:59.341035 23654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:59.341123 23654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:59.351402 23654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:40205,127.9.238.253:43311,127.9.238.252:45131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:40205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:46:59.352960 23654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:59.354341 23654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:59.368319 23659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:59.368435 23660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:59.368373 23662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:59.370146 23654 server_base.cc:1034] running on GCE node
I20260926 16:46:59.371109 23654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:59.372846 23654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:59.374081 23654 hybrid_clock.cc:648] HybridClock initialized: now 1790441219374016 us; error 56 us; skew 500 ppm
I20260926 16:46:59.374608 23654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:46:59 dist-test-slave-wt2j krb5kdc[23641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441219, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:46:59.378454 23654 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:46:59.380476 23654 webserver.cc:460] Webserver started at http://127.9.238.254:42241/ using document root <none> and password file <none>
I20260926 16:46:59.381379 23654 fs_manager.cc:359] Metadata directory not provided
I20260926 16:46:59.381524 23654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:46:59.381911 23654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:46:59.385242 23654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e"
format_stamp: "Formatted at 2026-09-26 16:46:59 on dist-test-slave-wt2j"
server_key: "6b352260f76eacb8e24b88e4975d81a3"
server_key_iv: "3c7fd628783d00f7b9023748727e3b5c"
server_key_version: "encryptionkey@0"
I20260926 16:46:59.386108 23654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e"
format_stamp: "Formatted at 2026-09-26 16:46:59 on dist-test-slave-wt2j"
server_key: "6b352260f76eacb8e24b88e4975d81a3"
server_key_iv: "3c7fd628783d00f7b9023748727e3b5c"
server_key_version: "encryptionkey@0"
I20260926 16:46:59.392251 23654 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260926 16:46:59.396818 23669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:46:59.398792 23654 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:46:59.399152 23654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e"
format_stamp: "Formatted at 2026-09-26 16:46:59 on dist-test-slave-wt2j"
server_key: "6b352260f76eacb8e24b88e4975d81a3"
server_key_iv: "3c7fd628783d00f7b9023748727e3b5c"
server_key_version: "encryptionkey@0"
I20260926 16:46:59.399381 23654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:46:59.422075 23654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:46:59.426950 23654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:46:59.427248 23654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:46:59.450615 23654 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:40205
I20260926 16:46:59.450749 23720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:40205 every 8 connection(s)
I20260926 16:46:59.452267 23654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:46:59.457231 23721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:46:59.458492 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 23654
I20260926 16:46:59.458751 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:46:59.459703 10171 external_mini_cluster.cc:1428] Setting key 411f084add448692c861a2cebd77ab89
I20260926 16:46:59.469340 23721 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40205 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.238.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.253@KRBTEST.COM" created.
Sep 26 16:46:59 dist-test-slave-wt2j krb5kdc[23641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441219, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:46:59.498606 23724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:59.474279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47728 (local address 127.9.238.254:40205)
0926 16:46:59.475409 (+ 1130us) server_negotiation.cc:206] Beginning negotiation
0926 16:46:59.475435 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 16:46:59.476023 (+ 588us) server_negotiation.cc:407] Connection header received
0926 16:46:59.478950 (+ 2927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:46:59.478959 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:46:59.479293 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:46:59.479589 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:46:59.481132 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:59.482556 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:59.484224 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:46:59.484921 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:46:59.485456 (+ 535us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:59.490797 (+ 5341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:46:59.490807 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:46:59.490822 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:46:59.490864 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:46:59.493939 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:59.494675 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:59.494681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:59.494684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:59.494832 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:46:59.495222 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:46:59.495228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:46:59.495236 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:46:59.495844 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:46:59.496103 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 16:46:59.496651 (+ 548us) server_negotiation.cc:299] Negotiation successful
0926 16:46:59.497064 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":822,"thread_start_us":536,"threads_started":1}
I20260926 16:46:59.498631 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:59.474296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40205 (local address 127.0.0.1:47728)
0926 16:46:59.475813 (+ 1517us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:59.475862 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 16:46:59.478110 (+ 2248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:46:59.479608 (+ 1498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:46:59.479617 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:46:59.480513 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:46:59.480911 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:59.480931 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:59.482738 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:59.482752 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:59.484061 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:46:59.484076 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:46:59.485511 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:46:59.485521 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:46:59.485741 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:46:59.486673 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:46:59.486706 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:46:59.490459 (+ 3753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:46:59.494110 (+ 3651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:59.494122 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:59.494134 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:59.494527 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:59.494943 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:46:59.494950 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:46:59.494953 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:46:59.495110 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:46:59.496027 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:46:59.496035 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:46:59.496265 (+ 230us) client_negotiation.cc:769] Sending connection context
0926 16:46:59.496512 (+ 247us) client_negotiation.cc:240] Negotiation successful
0926 16:46:59.497176 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1208,"mutex_wait_us":171,"thread_start_us":942,"threads_started":1}
I20260926 16:46:59.502197 23721 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.253" port: 43311 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:46:59.503968 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:59.503381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:43311 (local address 127.0.0.1:57094)
0926 16:46:59.503695 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:59.503828 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:43311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260926 16:46:59.504388 23672 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.238.253:43311: connect: Connection refused (error 111)
I20260926 16:46:59.506343 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:59.505954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:43311 (local address 127.0.0.1:57104)
0926 16:46:59.506108 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:59.506193 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:43311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 16:46:59.506573 23721 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.253:43311: Network error: Client connection negotiation failed: client connection to 127.9.238.253:43311: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.253.keytab.
Entry for principal kudu/127.9.238.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.253.keytab.
I20260926 16:46:59.524897 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.253:43311
--webserver_interface=127.9.238.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.253.keytab
--principal=kudu/127.9.238.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:40205,127.9.238.253:43311,127.9.238.252:45131
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:46:59.556026 23721 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.253" port: 43311 } attempt: 1
I20260926 16:46:59.557679 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:59.556891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:43311 (local address 127.0.0.1:57106)
0926 16:46:59.557216 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:59.557454 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:43311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 16:46:59.559084 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:59.558599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:43311 (local address 127.0.0.1:57116)
0926 16:46:59.558787 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:59.558888 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:43311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 16:46:59.559367 23721 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.253:43311: Network error: Client connection negotiation failed: client connection to 127.9.238.253:43311: connect: Connection refused (error 111)
I20260926 16:46:59.627727 23721 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.253" port: 43311 } attempt: 2
I20260926 16:46:59.629465 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:59.628689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:43311 (local address 127.0.0.1:57120)
0926 16:46:59.629053 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:59.629210 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:43311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260926 16:46:59.630940 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:59.630390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:43311 (local address 127.0.0.1:57132)
0926 16:46:59.630615 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:59.630735 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:43311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260926 16:46:59.631240 23721 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.253:43311: Network error: Client connection negotiation failed: client connection to 127.9.238.253:43311: connect: Connection refused (error 111)
I20260926 16:46:59.722610 23721 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.253" port: 43311 } attempt: 3
I20260926 16:46:59.724454 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:59.723442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:43311 (local address 127.0.0.1:57134)
0926 16:46:59.723957 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:59.724182 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:43311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260926 16:46:59.725927 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:59.725381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:43311 (local address 127.0.0.1:57150)
0926 16:46:59.725578 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:59.725697 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:43311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 16:46:59.726244 23721 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.253:43311: Network error: Client connection negotiation failed: client connection to 127.9.238.253:43311: connect: Connection refused (error 111)
I20260926 16:46:59.869553 23721 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.253" port: 43311 } attempt: 4
I20260926 16:46:59.871413 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:59.870471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:43311 (local address 127.0.0.1:57152)
0926 16:46:59.870986 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:59.871143 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:43311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277}
I20260926 16:46:59.872864 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:46:59.872305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:43311 (local address 127.0.0.1:57156)
0926 16:46:59.872504 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 16:46:59.872648 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:43311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 16:46:59.873147 23721 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.253:43311: Network error: Client connection negotiation failed: client connection to 127.9.238.253:43311: connect: Connection refused (error 111)
W20260926 16:46:59.947232 23728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:46:59.947656 23728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:46:59.947762 23728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:46:59.955961 23728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:46:59.956115 23728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:46:59.956164 23728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:46:59.956220 23728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:46:59.956279 23728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:46:59.968081 23728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:40205,127.9.238.253:43311,127.9.238.252:45131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.253.keytab
--principal=kudu/127.9.238.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.253:43311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.238.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:46:59.969853 23728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:46:59.971457 23728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:46:59.986609 23733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:59.986876 23734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:46:59.987031 23736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:46:59.987766 23728 server_base.cc:1034] running on GCE node
I20260926 16:46:59.988729 23728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:46:59.990407 23728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:46:59.991724 23728 hybrid_clock.cc:648] HybridClock initialized: now 1790441219991670 us; error 53 us; skew 500 ppm
I20260926 16:46:59.992245 23728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:47:00 dist-test-slave-wt2j krb5kdc[23641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441220, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:47:00.007380 23728 init.cc:377] Logged in from keytab as kudu/127.9.238.253@KRBTEST.COM (short username kudu)
I20260926 16:47:00.010684 23728 webserver.cc:460] Webserver started at http://127.9.238.253:34633/ using document root <none> and password file <none>
I20260926 16:47:00.012133 23728 fs_manager.cc:359] Metadata directory not provided
I20260926 16:47:00.012349 23728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:47:00.012861 23728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:47:00.017053 23728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-1/data/instance:
uuid: "a40483be77b149d79b3b688ed70c0d49"
format_stamp: "Formatted at 2026-09-26 16:47:00 on dist-test-slave-wt2j"
server_key: "6db9d87da9832773f565b51624c082a2"
server_key_iv: "b41618362e11a7b4fd34ab52260611c6"
server_key_version: "encryptionkey@0"
I20260926 16:47:00.018168 23728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-1/wal/instance:
uuid: "a40483be77b149d79b3b688ed70c0d49"
format_stamp: "Formatted at 2026-09-26 16:47:00 on dist-test-slave-wt2j"
server_key: "6db9d87da9832773f565b51624c082a2"
server_key_iv: "b41618362e11a7b4fd34ab52260611c6"
server_key_version: "encryptionkey@0"
I20260926 16:47:00.025672 23728 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260926 16:47:00.031972 23743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:00.034328 23728 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260926 16:47:00.034565 23728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-1/wal
uuid: "a40483be77b149d79b3b688ed70c0d49"
format_stamp: "Formatted at 2026-09-26 16:47:00 on dist-test-slave-wt2j"
server_key: "6db9d87da9832773f565b51624c082a2"
server_key_iv: "b41618362e11a7b4fd34ab52260611c6"
server_key_version: "encryptionkey@0"
I20260926 16:47:00.034816 23728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:47:00.056813 23728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:47:00.062196 23728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:47:00.062530 23728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:47:00.086215 23728 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.253:43311
I20260926 16:47:00.086266 23794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.253:43311 every 8 connection(s)
I20260926 16:47:00.088274 23728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-1/data/info.pb
I20260926 16:47:00.092983 23795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:47:00.097101 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 23728
I20260926 16:47:00.097374 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-1/wal/instance
I20260926 16:47:00.097955 10171 external_mini_cluster.cc:1428] Setting key 4793f25783a90d59df4f9f3c0eeaa888
I20260926 16:47:00.100457 23795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40205 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:47:00 dist-test-slave-wt2j krb5kdc[23641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441220, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.253@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.238.252@KRBTEST.COM; defaulting to no policy
I20260926 16:47:00.126295 23797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.104344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47736 (local address 127.9.238.254:40205)
0926 16:47:00.105016 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:00.105021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:00.106890 (+ 1869us) server_negotiation.cc:407] Connection header received
0926 16:47:00.109522 (+ 2632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:00.109528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:00.109657 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:00.109786 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:00.111677 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.112438 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.114059 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.114472 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.114610 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:00.119455 (+ 4845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:00.119461 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:00.119464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:00.119515 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:00.122635 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:00.123626 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:00.123631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:00.123633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:00.123768 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:00.124323 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:00.124327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:00.124329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:00.124638 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:00.124818 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:00.125630 (+ 812us) server_negotiation.cc:299] Negotiation successful
0926 16:47:00.125893 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":363,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.252@KRBTEST.COM" created.
I20260926 16:47:00.127467 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.105371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40205 (local address 127.0.0.1:47736)
0926 16:47:00.106605 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:00.106668 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:00.109085 (+ 2417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:00.110085 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:00.110096 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:00.110774 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:47:00.111443 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:00.111463 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.112614 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.112619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:00.113897 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:00.113910 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.114717 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.114724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:00.114950 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:00.115905 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:47:00.115931 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:47:00.119139 (+ 3208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:47:00.122846 (+ 3707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:00.122857 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:00.122886 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:00.123414 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:00.123970 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:00.123974 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:00.123976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:00.124176 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:00.124787 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:47:00.124796 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:47:00.125231 (+ 435us) client_negotiation.cc:769] Sending connection context
0926 16:47:00.125520 (+ 289us) client_negotiation.cc:240] Negotiation successful
0926 16:47:00.126075 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"thread_start_us":561,"threads_started":1}
I20260926 16:47:00.129772 23795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.253" port: 43311 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:47:00 dist-test-slave-wt2j krb5kdc[23641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441220, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.253@KRBTEST.COM for kudu/127.9.238.253@KRBTEST.COM
I20260926 16:47:00.150614 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.130887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:43311 (local address 127.0.0.1:57158)
0926 16:47:00.131076 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:00.131096 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:00.131222 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:00.133194 (+ 1972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:00.133199 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:00.133694 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:47:00.133887 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:00.133897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.136285 (+ 2388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.136290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:00.137462 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:00.137486 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.138781 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.138789 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:00.139096 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:00.140078 (+ 982us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:47:00.140109 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:47:00.143215 (+ 3106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:47:00.147609 (+ 4394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:00.147618 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:00.147623 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:00.148059 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:00.148492 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:00.148498 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:00.148504 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:00.148696 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:00.149469 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:47:00.149474 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:47:00.149634 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 16:47:00.149807 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 16:47:00.150162 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 16:47:00.150707 23800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.131235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57158 (local address 127.9.238.253:43311)
0926 16:47:00.132268 (+ 1033us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:00.132283 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:00.132341 (+ 58us) server_negotiation.cc:407] Connection header received
0926 16:47:00.132447 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:00.132463 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:00.132855 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:00.133127 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:00.134105 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.136031 (+ 1926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.137726 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.138534 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.138679 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:00.143611 (+ 4932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:00.143634 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:00.143662 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:00.143714 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:00.147392 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:00.148215 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:00.148231 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:00.148233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:00.148367 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:00.148843 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:00.148850 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:00.148853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:00.149317 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:00.149591 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:00.149974 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 16:47:00.150301 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":739,"thread_start_us":524,"threads_started":1}
I20260926 16:47:00.153971 23795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:47:00.155853 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.155163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:45131 (local address 127.0.0.1:42316)
0926 16:47:00.155438 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:00.155636 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 16:47:00.156227 23746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
I20260926 16:47:00.158407 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.157911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:45131 (local address 127.0.0.1:42320)
0926 16:47:00.158089 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:00.158236 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 16:47:00.158694 23795 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:45131: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.252.keytab.
Entry for principal kudu/127.9.238.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.252.keytab.
I20260926 16:47:00.165216 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.252:45131
--webserver_interface=127.9.238.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.252.keytab
--principal=kudu/127.9.238.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:40205,127.9.238.253:43311,127.9.238.252:45131
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:47:00.172461 23721 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.253" port: 43311 } attempt: 5
Sep 26 16:47:00 dist-test-slave-wt2j krb5kdc[23641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441219, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for kudu/127.9.238.253@KRBTEST.COM
I20260926 16:47:00.190106 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.173526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:43311 (local address 127.0.0.1:57164)
0926 16:47:00.173847 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:00.173877 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:00.174078 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:00.174576 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:00.174584 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:00.175161 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:47:00.175432 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:00.175447 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.176565 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.176573 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:00.178211 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:00.178229 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.179185 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.179193 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:00.179426 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:00.180348 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:47:00.180373 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:47:00.183158 (+ 2785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:47:00.187173 (+ 4015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:00.187180 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:00.187185 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:00.187756 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:00.188189 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:00.188194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:00.188198 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:00.188360 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:00.188969 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:47:00.188973 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:47:00.189158 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 16:47:00.189310 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 16:47:00.189714 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260926 16:47:00.190106 23800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.173709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57164 (local address 127.9.238.253:43311)
0926 16:47:00.173937 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:00.173942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:00.173967 (+ 25us) server_negotiation.cc:407] Connection header received
0926 16:47:00.174242 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:00.174247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:00.174341 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:00.174494 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:00.175662 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.176407 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.178395 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.178973 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.179145 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:00.183485 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:00.183491 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:00.183495 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:00.183538 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:00.186965 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:00.187907 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:00.187914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:00.187918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:00.188065 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:00.188486 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:00.188497 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:00.188499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:00.188842 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:00.189016 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:00.189451 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 16:47:00.189713 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 16:47:00.191764 23721 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:47:00.193241 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.192759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:45131 (local address 127.0.0.1:42322)
0926 16:47:00.192957 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:00.193056 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 16:47:00.194324 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.193977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:45131 (local address 127.0.0.1:42334)
0926 16:47:00.194124 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:00.194214 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 16:47:00.194559 23721 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:45131: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
I20260926 16:47:00.208187 23795 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } attempt: 1
I20260926 16:47:00.209664 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.209075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:45131 (local address 127.0.0.1:42340)
0926 16:47:00.209366 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:00.209483 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 16:47:00.210915 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.210535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:45131 (local address 127.0.0.1:42346)
0926 16:47:00.210716 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:00.210786 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 16:47:00.211196 23795 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:45131: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
I20260926 16:47:00.245895 23721 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } attempt: 1
I20260926 16:47:00.247623 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.246711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:45131 (local address 127.0.0.1:42348)
0926 16:47:00.247122 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:00.247296 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260926 16:47:00.248946 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.248509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:45131 (local address 127.0.0.1:42352)
0926 16:47:00.248691 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:00.248798 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 16:47:00.249321 23721 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:45131: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
I20260926 16:47:00.279484 23795 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } attempt: 2
I20260926 16:47:00.281152 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.280581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:45131 (local address 127.0.0.1:42368)
0926 16:47:00.280818 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:00.280944 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 16:47:00.282451 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.281998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:45131 (local address 127.0.0.1:42374)
0926 16:47:00.282168 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:00.282310 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 16:47:00.282681 23795 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:45131: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
I20260926 16:47:00.317659 23721 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } attempt: 2
I20260926 16:47:00.319463 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.318732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:45131 (local address 127.0.0.1:42380)
0926 16:47:00.319063 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:00.319190 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260926 16:47:00.320732 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.320336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:45131 (local address 127.0.0.1:42382)
0926 16:47:00.320507 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:00.320592 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260926 16:47:00.321038 23721 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:45131: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
I20260926 16:47:00.373971 23795 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } attempt: 3
I20260926 16:47:00.375602 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.374943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:45131 (local address 127.0.0.1:42398)
0926 16:47:00.375225 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:00.375338 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 16:47:00.377279 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.376691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:45131 (local address 127.0.0.1:42412)
0926 16:47:00.377024 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:00.377118 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200}
W20260926 16:47:00.377538 23795 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:45131: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
I20260926 16:47:00.427358 23721 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } attempt: 3
I20260926 16:47:00.429010 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.428392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:45131 (local address 127.0.0.1:42424)
0926 16:47:00.428647 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:00.428802 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 16:47:00.430210 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.429811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:45131 (local address 127.0.0.1:42426)
0926 16:47:00.430006 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:00.430086 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 16:47:00.430444 23721 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:45131: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
I20260926 16:47:00.520861 23795 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } attempt: 4
I20260926 16:47:00.522645 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.521896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:45131 (local address 127.0.0.1:42430)
0926 16:47:00.522245 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:00.522391 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260926 16:47:00.523950 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.523511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:45131 (local address 127.0.0.1:42446)
0926 16:47:00.523709 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:00.523814 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260926 16:47:00.524259 23795 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:45131: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
W20260926 16:47:00.544428 23803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:47:00.544737 23803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:47:00.544891 23803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:47:00.552448 23803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:47:00.552577 23803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:47:00.552624 23803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:47:00.552718 23803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:47:00.552791 23803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:47:00.563383 23803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:40205,127.9.238.253:43311,127.9.238.252:45131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.252.keytab
--principal=kudu/127.9.238.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.252:45131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.238.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:47:00.565002 23803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:47:00.566679 23803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:47:00.582399 23809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:00.582435 23811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:00.582512 23808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:47:00.583885 23803 server_base.cc:1034] running on GCE node
I20260926 16:47:00.584857 23803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:47:00.586524 23803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:47:00.587800 23803 hybrid_clock.cc:648] HybridClock initialized: now 1790441220587716 us; error 66 us; skew 500 ppm
I20260926 16:47:00.588220 23803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:47:00 dist-test-slave-wt2j krb5kdc[23641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441220, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:47:00.592028 23803 init.cc:377] Logged in from keytab as kudu/127.9.238.252@KRBTEST.COM (short username kudu)
I20260926 16:47:00.593770 23803 webserver.cc:460] Webserver started at http://127.9.238.252:35651/ using document root <none> and password file <none>
I20260926 16:47:00.594545 23803 fs_manager.cc:359] Metadata directory not provided
I20260926 16:47:00.594707 23803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:47:00.595052 23803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:47:00.598341 23803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-2/data/instance:
uuid: "417aa4de660d49268bd1a3db9c0956fd"
format_stamp: "Formatted at 2026-09-26 16:47:00 on dist-test-slave-wt2j"
server_key: "44e6b038949653de988cbbbc50c9aeae"
server_key_iv: "799e1b3d454435c01a3e97ac8ccea351"
server_key_version: "encryptionkey@0"
I20260926 16:47:00.599261 23803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-2/wal/instance:
uuid: "417aa4de660d49268bd1a3db9c0956fd"
format_stamp: "Formatted at 2026-09-26 16:47:00 on dist-test-slave-wt2j"
server_key: "44e6b038949653de988cbbbc50c9aeae"
server_key_iv: "799e1b3d454435c01a3e97ac8ccea351"
server_key_version: "encryptionkey@0"
I20260926 16:47:00.605592 23803 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20260926 16:47:00.607690 23721 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } attempt: 4
I20260926 16:47:00.609179 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.608547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:45131 (local address 127.0.0.1:42450)
0926 16:47:00.608849 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:00.608988 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 16:47:00.610134 23818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:00.610610 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.610095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:45131 (local address 127.0.0.1:42462)
0926 16:47:00.610330 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:00.610428 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260926 16:47:00.610916 23721 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:45131: Network error: Client connection negotiation failed: client connection to 127.9.238.252:45131: connect: Connection refused (error 111)
I20260926 16:47:00.612370 23803 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 16:47:00.612594 23803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-2/wal
uuid: "417aa4de660d49268bd1a3db9c0956fd"
format_stamp: "Formatted at 2026-09-26 16:47:00 on dist-test-slave-wt2j"
server_key: "44e6b038949653de988cbbbc50c9aeae"
server_key_iv: "799e1b3d454435c01a3e97ac8ccea351"
server_key_version: "encryptionkey@0"
I20260926 16:47:00.612815 23803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:47:00.643872 23803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:47:00.649288 23803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:47:00.649654 23803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:47:00.672597 23803 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.252:45131
I20260926 16:47:00.672662 23869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.252:45131 every 8 connection(s)
I20260926 16:47:00.674151 23803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-2/data/info.pb
I20260926 16:47:00.676824 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 23803
I20260926 16:47:00.677083 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790440988577176-10171-0/minicluster-data/master-2/wal/instance
I20260926 16:47:00.677843 10171 external_mini_cluster.cc:1428] Setting key 6ecc9a12bebc79f4b2a691967ae38484
I20260926 16:47:00.680773 23870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:47:00.692257 23870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40205 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:47:00 dist-test-slave-wt2j krb5kdc[23641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.252@KRBTEST.COM
I20260926 16:47:00.706408 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.680371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42464 (local address 127.9.238.252:45131)
0926 16:47:00.681433 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:00.681445 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:00.681494 (+ 49us) server_negotiation.cc:407] Connection header received
0926 16:47:00.683255 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:00.683279 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:00.683815 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:00.684220 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:00.686701 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.688419 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.690731 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.691706 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.691877 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:00.698416 (+ 6539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:00.698430 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:00.698454 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:00.698516 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:00.701839 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:00.702670 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:00.702677 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:00.702683 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:00.702864 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:00.703283 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:00.703291 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:00.703294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:00.703980 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:00.704324 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:00.704823 (+ 499us) server_negotiation.cc:299] Negotiation successful
0926 16:47:00.705294 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":825,"thread_start_us":547,"threads_started":1}
Sep 26 16:47:00 dist-test-slave-wt2j krb5kdc[23641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441220, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.252@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:47:00.715855 23875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.696851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40205 (local address 127.0.0.1:47746)
0926 16:47:00.698003 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:00.698062 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:00.698328 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:00.698906 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:00.698915 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:00.699946 (+ 1031us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:47:00.700371 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:00.700383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.701658 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.701668 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:00.703109 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:00.703123 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.704133 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.704139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:00.704354 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:00.705231 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:47:00.705262 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:47:00.708045 (+ 2783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:47:00.712539 (+ 4494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:00.712550 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:00.712557 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:00.713314 (+ 757us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:00.713800 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:00.713805 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:00.713809 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:00.714021 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:00.714708 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:47:00.714716 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:47:00.714910 (+ 194us) client_negotiation.cc:769] Sending connection context
0926 16:47:00.715098 (+ 188us) client_negotiation.cc:240] Negotiation successful
0926 16:47:00.715416 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"thread_start_us":691,"threads_started":1}
I20260926 16:47:00.715956 23874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.696823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47746 (local address 127.9.238.254:40205)
0926 16:47:00.697583 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:00.697593 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:00.698194 (+ 601us) server_negotiation.cc:407] Connection header received
0926 16:47:00.698588 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:00.698599 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:00.698695 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:00.698841 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:00.700621 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.701470 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.703267 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.703898 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.704070 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:00.708319 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:00.708325 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:00.708329 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:00.708371 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:00.712344 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:00.713490 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:00.713494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:00.713497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:00.713667 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:00.714135 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:00.714139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:00.714141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:00.714450 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:00.714749 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:00.715251 (+ 502us) server_negotiation.cc:299] Negotiation successful
0926 16:47:00.715512 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":398,"threads_started":1}
I20260926 16:47:00.718281 23870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.253" port: 43311 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:47:00 dist-test-slave-wt2j krb5kdc[23641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
Sep 26 16:47:00 dist-test-slave-wt2j krb5kdc[23641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441220, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.252@KRBTEST.COM for kudu/127.9.238.253@KRBTEST.COM
I20260926 16:47:00.730672 23876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.710287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47762 (local address 127.9.238.254:40205)
0926 16:47:00.711130 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:00.711136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:00.711159 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:47:00.711224 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:00.711228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:00.711299 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:00.711406 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:00.713176 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.714439 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.716485 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.717217 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.717365 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:00.722857 (+ 5492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:00.722862 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:00.722866 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:00.722899 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:00.726917 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:00.727984 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:00.727989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:00.727992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:00.728163 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:00.728652 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:00.728658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:00.728662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:00.729160 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:00.729537 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:00.729946 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 16:47:00.730220 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":425,"threads_started":1}
I20260926 16:47:00.738010 23875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.719207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:43311 (local address 127.0.0.1:57176)
0926 16:47:00.720642 (+ 1435us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:00.720658 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:00.720771 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:00.721221 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:00.721224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:00.721708 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:47:00.721877 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:00.721886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.723328 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.723334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:00.724761 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:00.724778 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.725979 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.726011 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:00.726309 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:00.726915 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:47:00.726933 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:47:00.729814 (+ 2881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:47:00.734915 (+ 5101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:00.734925 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:00.734928 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:00.735595 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:00.736142 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:00.736148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:00.736152 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:00.736353 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:00.736969 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:47:00.736977 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:47:00.737230 (+ 253us) client_negotiation.cc:769] Sending connection context
0926 16:47:00.737382 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 16:47:00.737723 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1308}
I20260926 16:47:00.738646 23877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.719307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57176 (local address 127.9.238.253:43311)
0926 16:47:00.719945 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:00.719951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:00.720787 (+ 836us) server_negotiation.cc:407] Connection header received
0926 16:47:00.720894 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:00.720900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:00.721015 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:00.721193 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:00.722078 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.723170 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.725035 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.725793 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.725932 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:00.730115 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:00.730122 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:00.730126 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:00.730298 (+ 172us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:00.734637 (+ 4339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:00.735749 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:00.735757 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:00.735760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:00.735944 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:00.736497 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:00.736501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:00.736504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:00.736820 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:00.736986 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:00.737552 (+ 566us) server_negotiation.cc:299] Negotiation successful
0926 16:47:00.737824 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":379,"threads_started":1}
I20260926 16:47:00.740080 23870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:47:00 dist-test-slave-wt2j krb5kdc[23641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.253@KRBTEST.COM
Sep 26 16:47:00 dist-test-slave-wt2j krb5kdc[23641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441220, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.252@KRBTEST.COM for kudu/127.9.238.252@KRBTEST.COM
I20260926 16:47:00.751065 23878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.732843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57186 (local address 127.9.238.253:43311)
0926 16:47:00.733659 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:00.733667 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:00.733695 (+ 28us) server_negotiation.cc:407] Connection header received
0926 16:47:00.733754 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:00.733758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:00.733838 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:00.733952 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:00.735681 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.737025 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.739009 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.739637 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.739786 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:00.745255 (+ 5469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:00.745261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:00.745264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:00.745297 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:00.748193 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:00.748953 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:00.748965 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:00.748969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:00.749102 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:00.749504 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:00.749508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:00.749511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:00.749805 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:00.750090 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:00.750415 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 16:47:00.750648 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":427,"threads_started":1}
I20260926 16:47:00.756209 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.741189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42466 (local address 127.9.238.252:45131)
0926 16:47:00.741370 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:00.741374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:00.741396 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:47:00.741699 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:00.741711 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:00.741776 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:00.741874 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:00.742928 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.744161 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.745447 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.745915 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.746074 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:00.749568 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:00.749573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:00.749577 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:00.749606 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:00.752773 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:00.753741 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:00.753756 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:00.753761 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:00.753978 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:00.754528 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:00.754535 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:00.754543 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:00.754862 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:00.755026 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:00.755399 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 16:47:00.755718 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 16:47:00.756265 23875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.741052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:45131 (local address 127.0.0.1:42466)
0926 16:47:00.741318 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:00.741345 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:00.741478 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:00.741953 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:00.741960 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:00.742512 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:47:00.742737 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:00.742749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.744335 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.744342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:00.745292 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:00.745303 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.746098 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.746104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:00.746326 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:00.746958 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:47:00.746977 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:47:00.749360 (+ 2383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:47:00.752915 (+ 3555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:00.752921 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:00.752926 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:00.753524 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:00.754145 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:00.754148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:00.754151 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:00.754400 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:00.754978 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:47:00.754983 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:47:00.755133 (+ 150us) client_negotiation.cc:769] Sending connection context
0926 16:47:00.755325 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 16:47:00.755900 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":202}
I20260926 16:47:00.757618 10171 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:47:00.762770 23870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 417aa4de660d49268bd1a3db9c0956fd: Bootstrap starting.
I20260926 16:47:00.768939 23870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 417aa4de660d49268bd1a3db9c0956fd: Neither blocks nor log segments found. Creating new log.
I20260926 16:47:00.771270 23870 log.cc:824] T 00000000000000000000000000000000 P 417aa4de660d49268bd1a3db9c0956fd: Log is configured to *not* fsync() on all Append() calls
I20260926 16:47:00.783313 23870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 417aa4de660d49268bd1a3db9c0956fd: No bootstrap required, opened a new log
I20260926 16:47:00.789251 23878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.763542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57188 (local address 127.9.238.253:43311)
0926 16:47:00.764099 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:00.764107 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:00.764853 (+ 746us) server_negotiation.cc:407] Connection header received
0926 16:47:00.765110 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:00.765117 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:00.765201 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:00.765274 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:00.768316 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.769553 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.772494 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.773225 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.773327 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:00.778840 (+ 5513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:00.778847 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:00.778853 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:00.778906 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:00.784972 (+ 6066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:00.786234 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:00.786243 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:00.786247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:00.786450 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:00.787044 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:00.787051 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:00.787056 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:00.787495 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:00.787909 (+ 414us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:00.788516 (+ 607us) server_negotiation.cc:299] Negotiation successful
0926 16:47:00.788863 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320}
I20260926 16:47:00.791350 23876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.763577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47766 (local address 127.9.238.254:40205)
0926 16:47:00.763859 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:00.763867 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:00.764686 (+ 819us) server_negotiation.cc:407] Connection header received
0926 16:47:00.765038 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:00.765047 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:00.765155 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:00.765243 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:00.768316 (+ 3073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.769541 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.772511 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.773210 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.773344 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:00.779878 (+ 6534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:00.779888 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:00.779892 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:00.779953 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:00.787101 (+ 7148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:00.788157 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:00.788164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:00.788172 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:00.788371 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:00.788917 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:00.788923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:00.788927 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:00.789377 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:00.789707 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:00.790609 (+ 902us) server_negotiation.cc:299] Negotiation successful
0926 16:47:00.790904 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 16:47:00.795769 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.765569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42480 (local address 127.9.238.252:45131)
0926 16:47:00.765769 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:00.765776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:00.775051 (+ 9275us) server_negotiation.cc:407] Connection header received
0926 16:47:00.775433 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:00.775441 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:00.775576 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:00.775663 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:00.778398 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.779725 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.782014 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.782803 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.782959 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:00.786981 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:00.786991 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:00.786998 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:00.787056 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:00.791906 (+ 4850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:00.793091 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:00.793097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:00.793102 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:00.793318 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:00.793802 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:00.793807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:00.793811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:00.794394 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:00.794682 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:00.795049 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 16:47:00.795322 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 16:47:00.796077 23870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 417aa4de660d49268bd1a3db9c0956fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40205 } } peers { permanent_uuid: "a40483be77b149d79b3b688ed70c0d49" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 43311 } } peers { permanent_uuid: "417aa4de660d49268bd1a3db9c0956fd" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } }
I20260926 16:47:00.796705 23870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 417aa4de660d49268bd1a3db9c0956fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:47:00.796859 23870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 417aa4de660d49268bd1a3db9c0956fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417aa4de660d49268bd1a3db9c0956fd, State: Initialized, Role: FOLLOWER
I20260926 16:47:00.797663 23870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 417aa4de660d49268bd1a3db9c0956fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40205 } } peers { permanent_uuid: "a40483be77b149d79b3b688ed70c0d49" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 43311 } } peers { permanent_uuid: "417aa4de660d49268bd1a3db9c0956fd" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } }
I20260926 16:47:00.798728 23887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 417aa4de660d49268bd1a3db9c0956fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40205 } } peers { permanent_uuid: "a40483be77b149d79b3b688ed70c0d49" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 43311 } } peers { permanent_uuid: "417aa4de660d49268bd1a3db9c0956fd" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } } }
I20260926 16:47:00.799263 23887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 417aa4de660d49268bd1a3db9c0956fd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:47:00.800405 23870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 417aa4de660d49268bd1a3db9c0956fd [sys.catalog]: configured and running, proceeding with master startup.
W20260926 16:47:00.813323 23898 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 417aa4de660d49268bd1a3db9c0956fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:47:00.813591 23898 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:47:00.810931 23803 master_runner.cc:186] Error getting master registration for 127.9.238.254:40205: OK, instance_id { permanent_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" instance_seqno: 1790441219443862 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 16:47:00.821341 23803 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 16:47:00.823518 23795 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } attempt: 5
Sep 26 16:47:00 dist-test-slave-wt2j krb5kdc[23641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441220, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.253@KRBTEST.COM for kudu/127.9.238.252@KRBTEST.COM
I20260926 16:47:00.842578 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.824385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:45131 (local address 127.0.0.1:42492)
0926 16:47:00.824698 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:00.824722 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:00.824865 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:00.825252 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:00.825258 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:00.825830 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:47:00.826063 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:00.826076 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.827337 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.827343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:00.829024 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:00.829041 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.830458 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.830465 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:00.830746 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:00.831478 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:47:00.831505 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:47:00.834387 (+ 2882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:47:00.838834 (+ 4447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:00.838841 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:00.838844 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:00.839666 (+ 822us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:00.840225 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:00.840229 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:00.840232 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:00.840463 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:00.841326 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:47:00.841334 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:47:00.841545 (+ 211us) client_negotiation.cc:769] Sending connection context
0926 16:47:00.841725 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 16:47:00.842096 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260926 16:47:00.843766 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.824424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42492 (local address 127.9.238.252:45131)
0926 16:47:00.824679 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:00.824685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:00.824809 (+ 124us) server_negotiation.cc:407] Connection header received
0926 16:47:00.825004 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:00.825008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:00.825081 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:00.825179 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:00.826227 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.827186 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.829247 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.830037 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.830191 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:00.834645 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:00.834656 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:00.834662 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:00.834706 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:00.838551 (+ 3845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:00.839837 (+ 1286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:00.839844 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:00.839847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:00.840083 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:00.840621 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:00.840626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:00.840630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:00.841070 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:00.841228 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:00.843086 (+ 1858us) server_negotiation.cc:299] Negotiation successful
0926 16:47:00.843373 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 16:47:00.849472 23795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a40483be77b149d79b3b688ed70c0d49: Bootstrap starting.
I20260926 16:47:00.854188 23795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a40483be77b149d79b3b688ed70c0d49: Neither blocks nor log segments found. Creating new log.
I20260926 16:47:00.855983 23795 log.cc:824] T 00000000000000000000000000000000 P a40483be77b149d79b3b688ed70c0d49: Log is configured to *not* fsync() on all Append() calls
I20260926 16:47:00.860121 23795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a40483be77b149d79b3b688ed70c0d49: No bootstrap required, opened a new log
I20260926 16:47:00.870121 23795 raft_consensus.cc:348] T 00000000000000000000000000000000 P a40483be77b149d79b3b688ed70c0d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40205 } } peers { permanent_uuid: "a40483be77b149d79b3b688ed70c0d49" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 43311 } } peers { permanent_uuid: "417aa4de660d49268bd1a3db9c0956fd" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } }
I20260926 16:47:00.870584 23795 raft_consensus.cc:374] T 00000000000000000000000000000000 P a40483be77b149d79b3b688ed70c0d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:47:00.870738 23795 raft_consensus.cc:729] T 00000000000000000000000000000000 P a40483be77b149d79b3b688ed70c0d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a40483be77b149d79b3b688ed70c0d49, State: Initialized, Role: FOLLOWER
I20260926 16:47:00.871517 23795 consensus_queue.cc:260] T 00000000000000000000000000000000 P a40483be77b149d79b3b688ed70c0d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40205 } } peers { permanent_uuid: "a40483be77b149d79b3b688ed70c0d49" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 43311 } } peers { permanent_uuid: "417aa4de660d49268bd1a3db9c0956fd" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } }
I20260926 16:47:00.872799 23901 sys_catalog.cc:455] T 00000000000000000000000000000000 P a40483be77b149d79b3b688ed70c0d49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40205 } } peers { permanent_uuid: "a40483be77b149d79b3b688ed70c0d49" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 43311 } } peers { permanent_uuid: "417aa4de660d49268bd1a3db9c0956fd" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } } }
I20260926 16:47:00.873250 23901 sys_catalog.cc:458] T 00000000000000000000000000000000 P a40483be77b149d79b3b688ed70c0d49 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:47:00.874410 23795 sys_catalog.cc:564] T 00000000000000000000000000000000 P a40483be77b149d79b3b688ed70c0d49 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:47:00.888161 23721 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } attempt: 5
W20260926 16:47:00.888749 23912 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a40483be77b149d79b3b688ed70c0d49: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:47:00.888931 23912 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:47:00.884675 23728 master_runner.cc:186] Error getting master registration for 127.9.238.254:40205: OK, instance_id { permanent_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" instance_seqno: 1790441219443862 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 16:47:00.900410 23728 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 16:47:00.902511 23824 tablet.cc:2369] T 00000000000000000000000000000000 P 417aa4de660d49268bd1a3db9c0956fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 16:47:00 dist-test-slave-wt2j krb5kdc[23641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441219, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for kudu/127.9.238.252@KRBTEST.COM
I20260926 16:47:00.912937 23873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.890220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42508 (local address 127.9.238.252:45131)
0926 16:47:00.890440 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:00.890446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:00.890470 (+ 24us) server_negotiation.cc:407] Connection header received
0926 16:47:00.890556 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:00.890562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:00.890663 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:00.890807 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:00.892220 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.893475 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.897663 (+ 4188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.898592 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.898741 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:00.904905 (+ 6164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:00.904914 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:00.904920 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:00.904975 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:00.909107 (+ 4132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:00.910267 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:00.910275 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:00.910280 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:00.910472 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:00.911003 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:00.911009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:00.911012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:00.911406 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:00.911608 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:00.912194 (+ 586us) server_negotiation.cc:299] Negotiation successful
0926 16:47:00.912463 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 16:47:00.912949 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:00.888975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:45131 (local address 127.0.0.1:42508)
0926 16:47:00.889287 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:00.889323 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:00.889502 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:00.891051 (+ 1549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:00.891058 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:00.891727 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:47:00.892012 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:00.892026 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.893692 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.893699 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:00.895358 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:00.895379 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:00.899032 (+ 3653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:00.899039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:00.899240 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:00.900215 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:47:00.900241 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:47:00.904563 (+ 4322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:47:00.909350 (+ 4787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:00.909358 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:00.909363 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:00.910075 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:00.910634 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:00.910640 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:00.910645 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:00.910862 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:00.911621 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:47:00.911627 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:47:00.911858 (+ 231us) client_negotiation.cc:769] Sending connection context
0926 16:47:00.912049 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 16:47:00.912453 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260926 16:47:00.924230 23721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e: Bootstrap starting.
I20260926 16:47:00.929610 23721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e: Neither blocks nor log segments found. Creating new log.
I20260926 16:47:00.931375 23721 log.cc:824] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e: Log is configured to *not* fsync() on all Append() calls
I20260926 16:47:00.938014 23721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e: No bootstrap required, opened a new log
I20260926 16:47:00.951095 23721 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40205 } } peers { permanent_uuid: "a40483be77b149d79b3b688ed70c0d49" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 43311 } } peers { permanent_uuid: "417aa4de660d49268bd1a3db9c0956fd" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } }
I20260926 16:47:00.951756 23721 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:47:00.951902 23721 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e, State: Initialized, Role: FOLLOWER
I20260926 16:47:00.952935 23721 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40205 } } peers { permanent_uuid: "a40483be77b149d79b3b688ed70c0d49" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 43311 } } peers { permanent_uuid: "417aa4de660d49268bd1a3db9c0956fd" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } }
I20260926 16:47:00.957154 23915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40205 } } peers { permanent_uuid: "a40483be77b149d79b3b688ed70c0d49" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 43311 } } peers { permanent_uuid: "417aa4de660d49268bd1a3db9c0956fd" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } } }
I20260926 16:47:00.957765 23915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:47:00.959055 23721 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:47:00.982959 23654 master_runner.cc:263] No leader master found from master 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e
I20260926 16:47:00.983536 23654 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 16:47:00.998742 23926 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:47:00.999034 23926 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:47:01.036209 23915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:47:01.036644 23915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40205 } } peers { permanent_uuid: "a40483be77b149d79b3b688ed70c0d49" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 43311 } } peers { permanent_uuid: "417aa4de660d49268bd1a3db9c0956fd" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } }
I20260926 16:47:01.038897 23915 leader_election.cc:288] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a40483be77b149d79b3b688ed70c0d49 (127.9.238.253:43311), 417aa4de660d49268bd1a3db9c0956fd (127.9.238.252:45131)
I20260926 16:47:01.040994 23845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "417aa4de660d49268bd1a3db9c0956fd" is_pre_election: true
I20260926 16:47:01.041934 23845 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 417aa4de660d49268bd1a3db9c0956fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e in term 0.
I20260926 16:47:01.043143 23770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a40483be77b149d79b3b688ed70c0d49" is_pre_election: true
I20260926 16:47:01.043867 23770 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a40483be77b149d79b3b688ed70c0d49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e in term 0.
I20260926 16:47:01.045126 23672 leader_election.cc:302] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 417aa4de660d49268bd1a3db9c0956fd, 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e; no voters:
I20260926 16:47:01.053294 23915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:47:01.053776 23915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:47:01.053994 23915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:47:01.058220 23915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40205 } } peers { permanent_uuid: "a40483be77b149d79b3b688ed70c0d49" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 43311 } } peers { permanent_uuid: "417aa4de660d49268bd1a3db9c0956fd" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } }
I20260926 16:47:01.060316 23770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a40483be77b149d79b3b688ed70c0d49"
I20260926 16:47:01.060604 23770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a40483be77b149d79b3b688ed70c0d49 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:47:01.064919 23770 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a40483be77b149d79b3b688ed70c0d49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e in term 1.
I20260926 16:47:01.065762 23672 leader_election.cc:302] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e, a40483be77b149d79b3b688ed70c0d49; no voters:
W20260926 16:47:01.066005 23749 tablet.cc:2369] T 00000000000000000000000000000000 P a40483be77b149d79b3b688ed70c0d49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:47:01.069494 23915 leader_election.cc:288] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e [CANDIDATE]: Term 1 election: Requested vote from peers a40483be77b149d79b3b688ed70c0d49 (127.9.238.253:43311), 417aa4de660d49268bd1a3db9c0956fd (127.9.238.252:45131)
I20260926 16:47:01.070318 23915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:47:01.070977 23915 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e [term 1 LEADER]: Becoming Leader. State: Replica: 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e, State: Running, Role: LEADER
I20260926 16:47:01.077338 23915 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40205 } } peers { permanent_uuid: "a40483be77b149d79b3b688ed70c0d49" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 43311 } } peers { permanent_uuid: "417aa4de660d49268bd1a3db9c0956fd" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } }
I20260926 16:47:01.084448 23845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "417aa4de660d49268bd1a3db9c0956fd"
I20260926 16:47:01.084792 23845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 417aa4de660d49268bd1a3db9c0956fd [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:47:01.089206 23845 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 417aa4de660d49268bd1a3db9c0956fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e in term 1.
I20260926 16:47:01.094482 23928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e. Latest consensus state: current_term: 1 leader_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40205 } } peers { permanent_uuid: "a40483be77b149d79b3b688ed70c0d49" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 43311 } } peers { permanent_uuid: "417aa4de660d49268bd1a3db9c0956fd" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } } }
I20260926 16:47:01.100956 23928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e [sys.catalog]: This master's current role is: LEADER
I20260926 16:47:01.108405 23930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:47:01.111928 23930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:47:01.136173 23845 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 417aa4de660d49268bd1a3db9c0956fd [term 1 FOLLOWER]: Refusing update from remote peer 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:47:01.138787 23770 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a40483be77b149d79b3b688ed70c0d49 [term 1 FOLLOWER]: Refusing update from remote peer 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:47:01.140609 23915 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "417aa4de660d49268bd1a3db9c0956fd" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:47:01.142328 23928 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a40483be77b149d79b3b688ed70c0d49" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 43311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:47:01.178144 23901 sys_catalog.cc:455] T 00000000000000000000000000000000 P a40483be77b149d79b3b688ed70c0d49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e. Latest consensus state: current_term: 1 leader_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40205 } } peers { permanent_uuid: "a40483be77b149d79b3b688ed70c0d49" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 43311 } } peers { permanent_uuid: "417aa4de660d49268bd1a3db9c0956fd" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } } }
I20260926 16:47:01.178623 23901 sys_catalog.cc:458] T 00000000000000000000000000000000 P a40483be77b149d79b3b688ed70c0d49 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 16:47:01.181322 23675 tablet.cc:2369] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:47:01.200212 23915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40205 } } peers { permanent_uuid: "a40483be77b149d79b3b688ed70c0d49" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 43311 } } peers { permanent_uuid: "417aa4de660d49268bd1a3db9c0956fd" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } } }
I20260926 16:47:01.200703 23915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e [sys.catalog]: This master's current role is: LEADER
I20260926 16:47:01.203279 23930 catalog_manager.cc:1347] Generated new cluster ID: c7f3994156f64d22953d39ab3b1e2988
I20260926 16:47:01.203482 23930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:47:01.204023 23901 sys_catalog.cc:455] T 00000000000000000000000000000000 P a40483be77b149d79b3b688ed70c0d49 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40205 } } peers { permanent_uuid: "a40483be77b149d79b3b688ed70c0d49" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 43311 } } peers { permanent_uuid: "417aa4de660d49268bd1a3db9c0956fd" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } } }
I20260926 16:47:01.204391 23901 sys_catalog.cc:458] T 00000000000000000000000000000000 P a40483be77b149d79b3b688ed70c0d49 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:47:01.211786 23887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 417aa4de660d49268bd1a3db9c0956fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e. Latest consensus state: current_term: 1 leader_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40205 } } peers { permanent_uuid: "a40483be77b149d79b3b688ed70c0d49" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 43311 } } peers { permanent_uuid: "417aa4de660d49268bd1a3db9c0956fd" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } } }
I20260926 16:47:01.212200 23887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 417aa4de660d49268bd1a3db9c0956fd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:47:01.215698 23928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40205 } } peers { permanent_uuid: "a40483be77b149d79b3b688ed70c0d49" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 43311 } } peers { permanent_uuid: "417aa4de660d49268bd1a3db9c0956fd" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } } }
I20260926 16:47:01.216073 23928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e [sys.catalog]: This master's current role is: LEADER
I20260926 16:47:01.220677 23933 mvcc.cc:205] Tried to move back new op lower bound from 7333647241752743936 to 7333647241546522624. Current Snapshot: MvccSnapshot[applied={T|T < 7333647241752743936}]
I20260926 16:47:01.228902 23887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 417aa4de660d49268bd1a3db9c0956fd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97269dcf2a6d4dc4ac0eb7c9ac6b9b2e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40205 } } peers { permanent_uuid: "a40483be77b149d79b3b688ed70c0d49" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 43311 } } peers { permanent_uuid: "417aa4de660d49268bd1a3db9c0956fd" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 45131 } } }
I20260926 16:47:01.229315 23887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 417aa4de660d49268bd1a3db9c0956fd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:47:01.238217 23930 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:47:01.240449 23930 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:47:01.279397 23930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97269dcf2a6d4dc4ac0eb7c9ac6b9b2e: Generated new TSK 0
I20260926 16:47:01.280675 23930 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:47:01.815896 23898 catalog_manager.cc:1259] Loaded cluster ID: c7f3994156f64d22953d39ab3b1e2988
I20260926 16:47:01.816017 23898 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 417aa4de660d49268bd1a3db9c0956fd: loading cluster ID for follower catalog manager: success
I20260926 16:47:01.819068 23898 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 417aa4de660d49268bd1a3db9c0956fd: acquiring CA information for follower catalog manager: success
I20260926 16:47:01.821229 23898 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 417aa4de660d49268bd1a3db9c0956fd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 16:47:01.891729 23912 catalog_manager.cc:1259] Loaded cluster ID: c7f3994156f64d22953d39ab3b1e2988
I20260926 16:47:01.891930 23912 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a40483be77b149d79b3b688ed70c0d49: loading cluster ID for follower catalog manager: success
I20260926 16:47:01.894853 23912 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a40483be77b149d79b3b688ed70c0d49: acquiring CA information for follower catalog manager: success
I20260926 16:47:01.897360 23912 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a40483be77b149d79b3b688ed70c0d49: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 16:47:01.919183 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 23654
I20260926 16:47:01.921077 23716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:47:02.039171 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 23654
I20260926 16:47:02.062511 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 23728
I20260926 16:47:02.064100 23790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:47:02.177925 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 23728
I20260926 16:47:02.202952 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 23803
I20260926 16:47:02.204384 23865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:47:02.324088 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 23803
2026-09-26T16:47:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4812 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:47:02 dist-test-slave-wt2j krb5kdc[23950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:47:02 dist-test-slave-wt2j krb5kdc[23950](info): set up 2 sockets
Sep 26 16:47:02 dist-test-slave-wt2j krb5kdc[23950](info): commencing operation
krb5kdc: starting...
W20260926 16:47:03.465353 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:47:03 dist-test-slave-wt2j krb5kdc[23950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:47:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:47:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:47:03.761684 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:33455
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:39567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:33455
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:47:04.198779 23963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:47:04.199208 23963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:47:04.199314 23963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:47:04.208400 23963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:47:04.208595 23963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:47:04.208650 23963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:47:04.208707 23963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:47:04.208760 23963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:47:04.220628 23963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:39567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:33455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:33455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:47:04.222470 23963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:47:04.224256 23963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:47:04.239456 23968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:04.239498 23969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:04.239594 23971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:47:04.241057 23963 server_base.cc:1034] running on GCE node
I20260926 16:47:04.242197 23963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:47:04.244390 23963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:47:04.245725 23963 hybrid_clock.cc:648] HybridClock initialized: now 1790441224245664 us; error 61 us; skew 500 ppm
I20260926 16:47:04.246327 23963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:47:04 dist-test-slave-wt2j krb5kdc[23950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441224, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:47:04.250644 23963 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:47:04.252889 23963 webserver.cc:460] Webserver started at http://127.9.238.254:39533/ using document root <none> and password file <none>
I20260926 16:47:04.253823 23963 fs_manager.cc:359] Metadata directory not provided
I20260926 16:47:04.254042 23963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:47:04.254473 23963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:47:04.258965 23963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "d97d105c59be4586bffe9e62ad3b08d4"
format_stamp: "Formatted at 2026-09-26 16:47:04 on dist-test-slave-wt2j"
server_key: "f84aed7ac9d53b629f938b3e8c7eaaca"
server_key_iv: "285d8f680dfe282fb3c19431d206826f"
server_key_version: "encryptionkey@0"
I20260926 16:47:04.260205 23963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "d97d105c59be4586bffe9e62ad3b08d4"
format_stamp: "Formatted at 2026-09-26 16:47:04 on dist-test-slave-wt2j"
server_key: "f84aed7ac9d53b629f938b3e8c7eaaca"
server_key_iv: "285d8f680dfe282fb3c19431d206826f"
server_key_version: "encryptionkey@0"
I20260926 16:47:04.267969 23963 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260926 16:47:04.273156 23978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:04.275532 23963 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260926 16:47:04.275904 23963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "d97d105c59be4586bffe9e62ad3b08d4"
format_stamp: "Formatted at 2026-09-26 16:47:04 on dist-test-slave-wt2j"
server_key: "f84aed7ac9d53b629f938b3e8c7eaaca"
server_key_iv: "285d8f680dfe282fb3c19431d206826f"
server_key_version: "encryptionkey@0"
I20260926 16:47:04.276212 23963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:47:04.292140 23963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:47:04.297860 23963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:47:04.298239 23963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:47:04.322996 23963 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:33455
I20260926 16:47:04.323052 24029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:33455 every 8 connection(s)
I20260926 16:47:04.324738 23963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:47:04.326661 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 23963
I20260926 16:47:04.326947 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:47:04.327886 10171 external_mini_cluster.cc:1428] Setting key d260c750e3ff1148b5b9a114a65480e0
I20260926 16:47:04.330734 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 16:47:04 dist-test-slave-wt2j krb5kdc[23950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:47:04.347435 24030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d97d105c59be4586bffe9e62ad3b08d4: Bootstrap starting.
I20260926 16:47:04.352957 24030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d97d105c59be4586bffe9e62ad3b08d4: Neither blocks nor log segments found. Creating new log.
I20260926 16:47:04.354770 24030 log.cc:824] T 00000000000000000000000000000000 P d97d105c59be4586bffe9e62ad3b08d4: Log is configured to *not* fsync() on all Append() calls
I20260926 16:47:04.359285 24030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d97d105c59be4586bffe9e62ad3b08d4: No bootstrap required, opened a new log
I20260926 16:47:04.359305 24033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:04.330596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36966 (local address 127.9.238.254:33455)
0926 16:47:04.332077 (+ 1481us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:04.332092 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:04.332143 (+ 51us) server_negotiation.cc:407] Connection header received
0926 16:47:04.334018 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:04.334044 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:04.334547 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:04.334966 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:04.337384 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:04.338923 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:04.341157 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:04.341987 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:04.342187 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:04.347927 (+ 5740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:04.347943 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:04.347971 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:04.348032 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:04.352578 (+ 4546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:04.353705 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:04.353716 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:04.353727 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:04.354026 (+ 299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:04.354813 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:04.354820 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:04.354822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:04.355987 (+ 1165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:04.356379 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:04.357174 (+ 795us) server_negotiation.cc:299] Negotiation successful
0926 16:47:04.357901 (+ 727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1214,"thread_start_us":904,"threads_started":1}
I20260926 16:47:04.365072 10171 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:47:04.369045 24030 raft_consensus.cc:348] T 00000000000000000000000000000000 P d97d105c59be4586bffe9e62ad3b08d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97d105c59be4586bffe9e62ad3b08d4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 33455 } }
I20260926 16:47:04.369586 24030 raft_consensus.cc:374] T 00000000000000000000000000000000 P d97d105c59be4586bffe9e62ad3b08d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:47:04.369697 24030 raft_consensus.cc:729] T 00000000000000000000000000000000 P d97d105c59be4586bffe9e62ad3b08d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d97d105c59be4586bffe9e62ad3b08d4, State: Initialized, Role: FOLLOWER
I20260926 16:47:04.370283 24030 consensus_queue.cc:260] T 00000000000000000000000000000000 P d97d105c59be4586bffe9e62ad3b08d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97d105c59be4586bffe9e62ad3b08d4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 33455 } }
I20260926 16:47:04.370530 24030 raft_consensus.cc:388] T 00000000000000000000000000000000 P d97d105c59be4586bffe9e62ad3b08d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:47:04.370671 24030 raft_consensus.cc:482] T 00000000000000000000000000000000 P d97d105c59be4586bffe9e62ad3b08d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:47:04.370862 24030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d97d105c59be4586bffe9e62ad3b08d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:47:04.373265 24030 raft_consensus.cc:504] T 00000000000000000000000000000000 P d97d105c59be4586bffe9e62ad3b08d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97d105c59be4586bffe9e62ad3b08d4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 33455 } }
I20260926 16:47:04.373814 24030 leader_election.cc:302] T 00000000000000000000000000000000 P d97d105c59be4586bffe9e62ad3b08d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d97d105c59be4586bffe9e62ad3b08d4; no voters:
I20260926 16:47:04.374514 24030 leader_election.cc:288] T 00000000000000000000000000000000 P d97d105c59be4586bffe9e62ad3b08d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:47:04.374800 24035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d97d105c59be4586bffe9e62ad3b08d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:47:04.375967 24035 raft_consensus.cc:686] T 00000000000000000000000000000000 P d97d105c59be4586bffe9e62ad3b08d4 [term 1 LEADER]: Becoming Leader. State: Replica: d97d105c59be4586bffe9e62ad3b08d4, State: Running, Role: LEADER
I20260926 16:47:04.376695 24035 consensus_queue.cc:237] T 00000000000000000000000000000000 P d97d105c59be4586bffe9e62ad3b08d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97d105c59be4586bffe9e62ad3b08d4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 33455 } }
I20260926 16:47:04.377112 24030 sys_catalog.cc:564] T 00000000000000000000000000000000 P d97d105c59be4586bffe9e62ad3b08d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:47:04.380613 24036 sys_catalog.cc:455] T 00000000000000000000000000000000 P d97d105c59be4586bffe9e62ad3b08d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d97d105c59be4586bffe9e62ad3b08d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97d105c59be4586bffe9e62ad3b08d4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 33455 } } }
I20260926 16:47:04.381165 24036 sys_catalog.cc:458] T 00000000000000000000000000000000 P d97d105c59be4586bffe9e62ad3b08d4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:47:04.381824 24037 sys_catalog.cc:455] T 00000000000000000000000000000000 P d97d105c59be4586bffe9e62ad3b08d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d97d105c59be4586bffe9e62ad3b08d4. Latest consensus state: current_term: 1 leader_uuid: "d97d105c59be4586bffe9e62ad3b08d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d97d105c59be4586bffe9e62ad3b08d4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 33455 } } }
I20260926 16:47:04.382686 24037 sys_catalog.cc:458] T 00000000000000000000000000000000 P d97d105c59be4586bffe9e62ad3b08d4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:47:04.384819 24041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:47:04.391881 24041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:47:04.404780 24041 catalog_manager.cc:1347] Generated new cluster ID: fff2765e5d884110bd2daa8e4c88e1eb
I20260926 16:47:04.405009 24041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:47:04.422590 24041 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:47:04.424147 24041 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:47:04.437325 24041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d97d105c59be4586bffe9e62ad3b08d4: Generated new TSK 0
I20260926 16:47:04.438346 24041 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:47:04.478382 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 23963
I20260926 16:47:04.479688 24025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:47:04.590843 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 23963
2026-09-26T16:47:04Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2262 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:47:04 dist-test-slave-wt2j krb5kdc[24058](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:47:04 dist-test-slave-wt2j krb5kdc[24058](info): set up 2 sockets
Sep 26 16:47:04 dist-test-slave-wt2j krb5kdc[24058](info): commencing operation
krb5kdc: starting...
W20260926 16:47:05.754175 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.005s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:47:05 dist-test-slave-wt2j krb5kdc[24058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:47:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:47:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:47:06.048869 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:46415
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:36335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:46415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:47:06.456339 24071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:47:06.456751 24071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:47:06.457055 24071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:47:06.464886 24071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:47:06.465170 24071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:47:06.465291 24071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:47:06.465404 24071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:47:06.465523 24071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:47:06.476377 24071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:36335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:46415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:46415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:47:06.477885 24071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:47:06.479384 24071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:47:06.495085 24079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:06.495182 24077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:06.495595 24076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:47:06.497071 24071 server_base.cc:1034] running on GCE node
I20260926 16:47:06.498113 24071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:47:06.499991 24071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:47:06.501353 24071 hybrid_clock.cc:648] HybridClock initialized: now 1790441226501269 us; error 87 us; skew 500 ppm
I20260926 16:47:06.501808 24071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:47:06 dist-test-slave-wt2j krb5kdc[24058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441226, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:47:06.505488 24071 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:47:06.507421 24071 webserver.cc:460] Webserver started at http://127.9.238.254:34423/ using document root <none> and password file <none>
I20260926 16:47:06.508279 24071 fs_manager.cc:359] Metadata directory not provided
I20260926 16:47:06.508397 24071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:47:06.508664 24071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:47:06.512246 24071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "2a81ad67dca84ffabddd19f5d60e3cad"
format_stamp: "Formatted at 2026-09-26 16:47:06 on dist-test-slave-wt2j"
server_key: "c419107dd2a6a00cced3eb5c7ff9c412"
server_key_iv: "049fca501495e8e60d3582a1b40ec1c4"
server_key_version: "encryptionkey@0"
I20260926 16:47:06.513283 24071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "2a81ad67dca84ffabddd19f5d60e3cad"
format_stamp: "Formatted at 2026-09-26 16:47:06 on dist-test-slave-wt2j"
server_key: "c419107dd2a6a00cced3eb5c7ff9c412"
server_key_iv: "049fca501495e8e60d3582a1b40ec1c4"
server_key_version: "encryptionkey@0"
I20260926 16:47:06.519534 24071 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.006s
I20260926 16:47:06.524106 24086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:06.525982 24071 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260926 16:47:06.526245 24071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "2a81ad67dca84ffabddd19f5d60e3cad"
format_stamp: "Formatted at 2026-09-26 16:47:06 on dist-test-slave-wt2j"
server_key: "c419107dd2a6a00cced3eb5c7ff9c412"
server_key_iv: "049fca501495e8e60d3582a1b40ec1c4"
server_key_version: "encryptionkey@0"
I20260926 16:47:06.526477 24071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:47:06.549638 24071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:47:06.554917 24071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:47:06.555380 24071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:47:06.579221 24071 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:46415
I20260926 16:47:06.579331 24137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:46415 every 8 connection(s)
I20260926 16:47:06.580870 24071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:47:06.581296 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 24071
I20260926 16:47:06.581545 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:47:06.582504 10171 external_mini_cluster.cc:1428] Setting key ee333a57f88c8a26e4f9c17655d3ee38
I20260926 16:47:06.587258 24138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 16:47:06 dist-test-slave-wt2j krb5kdc[24058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:47:06.604067 24138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad: Bootstrap starting.
I20260926 16:47:06.609280 24138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad: Neither blocks nor log segments found. Creating new log.
I20260926 16:47:06.611003 24138 log.cc:824] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad: Log is configured to *not* fsync() on all Append() calls
I20260926 16:47:06.612078 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:06.584956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45640 (local address 127.9.238.254:46415)
0926 16:47:06.585959 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:06.585979 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:06.586047 (+ 68us) server_negotiation.cc:407] Connection header received
0926 16:47:06.588668 (+ 2621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:06.588720 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:06.589382 (+ 662us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:06.590160 (+ 778us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:06.591710 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:06.594005 (+ 2295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:06.595466 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:06.596733 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:06.596990 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:06.601457 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:06.601472 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:06.601505 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:06.601570 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:06.606205 (+ 4635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:06.607209 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:06.607220 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:06.607231 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:06.607500 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:06.607959 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:06.607964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:06.607967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:06.608856 (+ 889us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:06.609262 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:06.609680 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 16:47:06.610310 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":382,"threads_started":1}
I20260926 16:47:06.615082 24138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad: No bootstrap required, opened a new log
I20260926 16:47:06.625638 24138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a81ad67dca84ffabddd19f5d60e3cad" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46415 } }
I20260926 16:47:06.626298 24138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:47:06.626447 24138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a81ad67dca84ffabddd19f5d60e3cad, State: Initialized, Role: FOLLOWER
I20260926 16:47:06.627377 24138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a81ad67dca84ffabddd19f5d60e3cad" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46415 } }
I20260926 16:47:06.627858 24138 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:47:06.628095 24138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:47:06.628376 24138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:47:06.631628 24138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a81ad67dca84ffabddd19f5d60e3cad" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46415 } }
I20260926 16:47:06.632376 24138 leader_election.cc:302] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a81ad67dca84ffabddd19f5d60e3cad; no voters:
I20260926 16:47:06.633267 24138 leader_election.cc:288] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:47:06.633510 24143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:47:06.634428 24143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [term 1 LEADER]: Becoming Leader. State: Replica: 2a81ad67dca84ffabddd19f5d60e3cad, State: Running, Role: LEADER
I20260926 16:47:06.635135 24143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a81ad67dca84ffabddd19f5d60e3cad" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46415 } }
I20260926 16:47:06.636232 24138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:47:06.639165 24144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a81ad67dca84ffabddd19f5d60e3cad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a81ad67dca84ffabddd19f5d60e3cad" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46415 } } }
I20260926 16:47:06.639202 24145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a81ad67dca84ffabddd19f5d60e3cad. Latest consensus state: current_term: 1 leader_uuid: "2a81ad67dca84ffabddd19f5d60e3cad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a81ad67dca84ffabddd19f5d60e3cad" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46415 } } }
I20260926 16:47:06.639706 24144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [sys.catalog]: This master's current role is: LEADER
I20260926 16:47:06.639756 24145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [sys.catalog]: This master's current role is: LEADER
I20260926 16:47:06.646214 24148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:47:06.654328 24148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:47:06.664223 24148 catalog_manager.cc:1347] Generated new cluster ID: 58ed3eb0b84b4b2b804dc006c690be83
I20260926 16:47:06.664361 24148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:47:06.679790 24148 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:47:06.681174 24148 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:47:06.699239 24148 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad: Generated new TSK 0
I20260926 16:47:06.700287 24148 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
I20260926 16:47:06.774600 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:0
--local_ip_for_outbound_sockets=127.9.238.193
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:46415
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:36335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:47:07.153328 24164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:47:07.153683 24164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:47:07.153853 24164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:47:07.161567 24164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:47:07.161783 24164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:47:07.161991 24164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:47:07.172410 24164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:36335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:46415
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:47:07.173838 24164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:47:07.175225 24164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:47:07.192118 24172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:07.192118 24170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:07.192214 24169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:47:07.199906 24164 server_base.cc:1034] running on GCE node
I20260926 16:47:07.200928 24164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:47:07.202306 24164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:47:07.203666 24164 hybrid_clock.cc:648] HybridClock initialized: now 1790441227203565 us; error 72 us; skew 500 ppm
I20260926 16:47:07.204144 24164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:47:07 dist-test-slave-wt2j krb5kdc[24058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441227, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:47:07.209089 24164 init.cc:377] Logged in from keytab as kudu/127.9.238.193@KRBTEST.COM (short username kudu)
I20260926 16:47:07.211359 24164 webserver.cc:460] Webserver started at http://127.9.238.193:39237/ using document root <none> and password file <none>
I20260926 16:47:07.212464 24164 fs_manager.cc:359] Metadata directory not provided
I20260926 16:47:07.212635 24164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:47:07.213076 24164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:47:07.217079 24164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/data/instance:
uuid: "9a83f18a7e554e8ba00204ce1550a273"
format_stamp: "Formatted at 2026-09-26 16:47:07 on dist-test-slave-wt2j"
server_key: "6d01d3f70319ed0ddcf6826a5ce62ace"
server_key_iv: "95742a9c60913135bbf7407dd93a1092"
server_key_version: "encryptionkey@0"
I20260926 16:47:07.218099 24164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance:
uuid: "9a83f18a7e554e8ba00204ce1550a273"
format_stamp: "Formatted at 2026-09-26 16:47:07 on dist-test-slave-wt2j"
server_key: "6d01d3f70319ed0ddcf6826a5ce62ace"
server_key_iv: "95742a9c60913135bbf7407dd93a1092"
server_key_version: "encryptionkey@0"
I20260926 16:47:07.225339 24164 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260926 16:47:07.230557 24179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:07.232777 24164 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 16:47:07.233088 24164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "9a83f18a7e554e8ba00204ce1550a273"
format_stamp: "Formatted at 2026-09-26 16:47:07 on dist-test-slave-wt2j"
server_key: "6d01d3f70319ed0ddcf6826a5ce62ace"
server_key_iv: "95742a9c60913135bbf7407dd93a1092"
server_key_version: "encryptionkey@0"
I20260926 16:47:07.233383 24164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:47:07.255064 24164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:47:07.260900 24164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:47:07.261291 24164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:47:07.262601 24164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:47:07.264475 24164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:47:07.264588 24164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:07.264779 24164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:47:07.264854 24164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:07.307626 24164 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:45243
I20260926 16:47:07.307696 24291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:45243 every 8 connection(s)
I20260926 16:47:07.309391 24164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:47:07.318631 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 24164
I20260926 16:47:07.318929 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance
I20260926 16:47:07.319738 10171 external_mini_cluster.cc:1428] Setting key 472bf9dd2933c727f6dca84076cc00e4
Sep 26 16:47:07 dist-test-slave-wt2j krb5kdc[24058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441227, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:47:07.335980 24294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:07.312796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:49611 (local address 127.9.238.254:46415)
0926 16:47:07.313321 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:07.313326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:07.314684 (+ 1358us) server_negotiation.cc:407] Connection header received
0926 16:47:07.316782 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:07.316787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:07.316885 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:07.317010 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:07.318669 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:07.319520 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:07.321440 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:07.321884 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:07.322050 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:07.327229 (+ 5179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:07.327235 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:07.327238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:07.327273 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:07.331100 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:07.332329 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:07.332340 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:07.332347 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:07.332652 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:07.333314 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:07.333322 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:07.333328 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:07.333783 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:07.333961 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:07.335096 (+ 1135us) server_negotiation.cc:299] Negotiation successful
0926 16:47:07.335503 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":237,"threads_started":1}
I20260926 16:47:07.337268 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:07.313583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.193:49611)
0926 16:47:07.314469 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:07.314522 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:07.316475 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:07.317276 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:07.317285 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:07.317859 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:47:07.318481 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:07.318499 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:07.319750 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:07.319754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:07.321281 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:07.321294 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:07.322239 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:07.322249 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:07.322670 (+ 421us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:07.323645 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:47:07.323691 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:47:07.326961 (+ 3270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:47:07.331351 (+ 4390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:07.331364 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:07.331386 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:07.332124 (+ 738us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:07.332847 (+ 723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:07.332854 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:07.332860 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:07.333138 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:07.333951 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:47:07.333962 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:47:07.334518 (+ 556us) client_negotiation.cc:769] Sending connection context
0926 16:47:07.334879 (+ 361us) client_negotiation.cc:240] Negotiation successful
0926 16:47:07.335539 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":346,"threads_started":1}
I20260926 16:47:07.340183 24292 heartbeater.cc:347] Connected to a master server at 127.9.238.254:46415
I20260926 16:47:07.341024 24292 heartbeater.cc:464] Registering TS with master...
I20260926 16:47:07.342176 24292 heartbeater.cc:511] Master 127.9.238.254:46415 requested a full tablet report, sending...
I20260926 16:47:07.345811 24103 ts_manager.cc:194] Registered new tserver with Master: 9a83f18a7e554e8ba00204ce1550a273 (127.9.238.193:45243)
I20260926 16:47:07.348507 24103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.193@KRBTEST.COM'} at 127.9.238.193:49611
WARNING: no policy specified for kudu/127.9.238.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
I20260926 16:47:07.392772 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:0
--local_ip_for_outbound_sockets=127.9.238.194
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:46415
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:36335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:47:07.814551 24298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:47:07.814942 24298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:47:07.815127 24298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:47:07.823652 24298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:47:07.823812 24298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:47:07.824013 24298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:47:07.835639 24298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:36335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:46415
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:47:07.837559 24298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:47:07.839375 24298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:47:07.857453 24303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:07.857638 24304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:07.857888 24306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:47:07.859186 24298 server_base.cc:1034] running on GCE node
I20260926 16:47:07.860046 24298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:47:07.861348 24298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:47:07.862644 24298 hybrid_clock.cc:648] HybridClock initialized: now 1790441227862577 us; error 60 us; skew 500 ppm
I20260926 16:47:07.863062 24298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:47:07 dist-test-slave-wt2j krb5kdc[24058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441227, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:47:07.867065 24298 init.cc:377] Logged in from keytab as kudu/127.9.238.194@KRBTEST.COM (short username kudu)
I20260926 16:47:07.869309 24298 webserver.cc:460] Webserver started at http://127.9.238.194:40805/ using document root <none> and password file <none>
I20260926 16:47:07.870265 24298 fs_manager.cc:359] Metadata directory not provided
I20260926 16:47:07.870443 24298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:47:07.870847 24298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:47:07.874647 24298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/data/instance:
uuid: "0685cb7ee46548989fb7a170b111181a"
format_stamp: "Formatted at 2026-09-26 16:47:07 on dist-test-slave-wt2j"
server_key: "1691e2ee85552ffcd7a026cacd9a0501"
server_key_iv: "ecb1b09a3cb01e428f4cde3b2287f1f0"
server_key_version: "encryptionkey@0"
I20260926 16:47:07.875681 24298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance:
uuid: "0685cb7ee46548989fb7a170b111181a"
format_stamp: "Formatted at 2026-09-26 16:47:07 on dist-test-slave-wt2j"
server_key: "1691e2ee85552ffcd7a026cacd9a0501"
server_key_iv: "ecb1b09a3cb01e428f4cde3b2287f1f0"
server_key_version: "encryptionkey@0"
I20260926 16:47:07.882699 24298 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260926 16:47:07.887616 24313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:07.889539 24298 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260926 16:47:07.889793 24298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "0685cb7ee46548989fb7a170b111181a"
format_stamp: "Formatted at 2026-09-26 16:47:07 on dist-test-slave-wt2j"
server_key: "1691e2ee85552ffcd7a026cacd9a0501"
server_key_iv: "ecb1b09a3cb01e428f4cde3b2287f1f0"
server_key_version: "encryptionkey@0"
I20260926 16:47:07.890050 24298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:47:07.923110 24298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:47:07.928361 24298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:47:07.928726 24298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:47:07.929909 24298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:47:07.931876 24298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:47:07.932006 24298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:07.932179 24298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:47:07.932263 24298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:07.975749 24298 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:35439
I20260926 16:47:07.975801 24425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:35439 every 8 connection(s)
I20260926 16:47:07.977309 24298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:47:07.984279 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 24298
I20260926 16:47:07.984539 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance
I20260926 16:47:07.985126 10171 external_mini_cluster.cc:1428] Setting key 3cbbc8c4af7f05d6fd8a0ce0e7b02f2b
Sep 26 16:47:07 dist-test-slave-wt2j krb5kdc[24058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441227, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:47:08.006418 24428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:07.981086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:45627 (local address 127.9.238.254:46415)
0926 16:47:07.981683 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:07.981689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:07.982996 (+ 1307us) server_negotiation.cc:407] Connection header received
0926 16:47:07.985421 (+ 2425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:07.985427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:07.985535 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:07.985671 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:07.987733 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:07.988946 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:07.991110 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:07.992013 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:07.992193 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:07.997166 (+ 4973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:07.997172 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:07.997176 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:07.997221 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:08.001543 (+ 4322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:08.003088 (+ 1545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:08.003113 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:08.003119 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:08.003323 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:08.004055 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:08.004061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:08.004064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:08.004519 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:08.004719 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:08.005618 (+ 899us) server_negotiation.cc:299] Negotiation successful
0926 16:47:08.005902 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":306,"threads_started":1}
I20260926 16:47:08.007946 24429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:07.981778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.194:45627)
0926 16:47:07.982783 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:07.982842 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:07.985060 (+ 2218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:07.985975 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:07.985985 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:07.986753 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:47:07.987500 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:07.987528 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:07.989184 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:07.989191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:07.990892 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:07.990918 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:07.992308 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:07.992314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:07.992591 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:07.993593 (+ 1002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:47:07.993625 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:47:07.996822 (+ 3197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:47:08.001820 (+ 4998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:08.001833 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:08.001883 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:08.002919 (+ 1036us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:08.003461 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:08.003467 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:08.003472 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:08.003881 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:08.004657 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:47:08.004667 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:47:08.005110 (+ 443us) client_negotiation.cc:769] Sending connection context
0926 16:47:08.005496 (+ 386us) client_negotiation.cc:240] Negotiation successful
0926 16:47:08.006147 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":393,"threads_started":1}
I20260926 16:47:08.010994 24426 heartbeater.cc:347] Connected to a master server at 127.9.238.254:46415
I20260926 16:47:08.011404 24426 heartbeater.cc:464] Registering TS with master...
I20260926 16:47:08.012579 24426 heartbeater.cc:511] Master 127.9.238.254:46415 requested a full tablet report, sending...
I20260926 16:47:08.014739 24103 ts_manager.cc:194] Registered new tserver with Master: 0685cb7ee46548989fb7a170b111181a (127.9.238.194:35439)
I20260926 16:47:08.016382 24103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.194@KRBTEST.COM'} at 127.9.238.194:45627
WARNING: no policy specified for kudu/127.9.238.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
I20260926 16:47:08.060981 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:0
--local_ip_for_outbound_sockets=127.9.238.195
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:46415
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:36335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:47:08.353672 24292 heartbeater.cc:503] Master 127.9.238.254:46415 was elected leader, sending a full tablet report...
W20260926 16:47:08.483436 24432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:47:08.483902 24432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:47:08.484079 24432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:47:08.492134 24432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:47:08.492300 24432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:47:08.492512 24432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:47:08.503137 24432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:36335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:46415
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:47:08.504726 24432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:47:08.506325 24432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:47:08.522858 24437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:08.522936 24440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:08.522917 24438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:47:08.529353 24432 server_base.cc:1034] running on GCE node
I20260926 16:47:08.530269 24432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:47:08.531527 24432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:47:08.532847 24432 hybrid_clock.cc:648] HybridClock initialized: now 1790441228532784 us; error 53 us; skew 500 ppm
I20260926 16:47:08.533308 24432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:47:08 dist-test-slave-wt2j krb5kdc[24058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441228, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:47:08.537393 24432 init.cc:377] Logged in from keytab as kudu/127.9.238.195@KRBTEST.COM (short username kudu)
I20260926 16:47:08.540007 24432 webserver.cc:460] Webserver started at http://127.9.238.195:36199/ using document root <none> and password file <none>
I20260926 16:47:08.540931 24432 fs_manager.cc:359] Metadata directory not provided
I20260926 16:47:08.541100 24432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:47:08.541457 24432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:47:08.544889 24432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/data/instance:
uuid: "fde78bbc9fed462a855a04094fde18da"
format_stamp: "Formatted at 2026-09-26 16:47:08 on dist-test-slave-wt2j"
server_key: "9ac1b971a5e4869a36e58713887f9265"
server_key_iv: "f3887687ab9409bb9fddff75e8066089"
server_key_version: "encryptionkey@0"
I20260926 16:47:08.545746 24432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance:
uuid: "fde78bbc9fed462a855a04094fde18da"
format_stamp: "Formatted at 2026-09-26 16:47:08 on dist-test-slave-wt2j"
server_key: "9ac1b971a5e4869a36e58713887f9265"
server_key_iv: "f3887687ab9409bb9fddff75e8066089"
server_key_version: "encryptionkey@0"
I20260926 16:47:08.551887 24432 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 16:47:08.556080 24447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:08.557914 24432 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 16:47:08.558212 24432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "fde78bbc9fed462a855a04094fde18da"
format_stamp: "Formatted at 2026-09-26 16:47:08 on dist-test-slave-wt2j"
server_key: "9ac1b971a5e4869a36e58713887f9265"
server_key_iv: "f3887687ab9409bb9fddff75e8066089"
server_key_version: "encryptionkey@0"
I20260926 16:47:08.558449 24432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:47:08.582231 24432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:47:08.587217 24432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:47:08.587620 24432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:47:08.589010 24432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:47:08.590605 24432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:47:08.590720 24432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:08.590885 24432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:47:08.590957 24432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:08.632383 24432 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:38291
I20260926 16:47:08.632421 24559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:38291 every 8 connection(s)
I20260926 16:47:08.633751 24432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:47:08.639843 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 24432
I20260926 16:47:08.640071 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance
I20260926 16:47:08.640609 10171 external_mini_cluster.cc:1428] Setting key b0eb935b8fceacb01ccfad39a255b84f
Sep 26 16:47:08 dist-test-slave-wt2j krb5kdc[24058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441228, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:47:08.659461 24562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:08.637190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:41561 (local address 127.9.238.254:46415)
0926 16:47:08.637839 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:08.637845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:08.639087 (+ 1242us) server_negotiation.cc:407] Connection header received
0926 16:47:08.641517 (+ 2430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:08.641526 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:08.641650 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:08.641847 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:08.644141 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:08.645288 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:08.647263 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:08.647849 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:08.647982 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:08.652083 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:08.652090 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:08.652094 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:08.652143 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:08.655589 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:08.656708 (+ 1119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:08.656716 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:08.656721 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:08.656962 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:08.657560 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:08.657565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:08.657567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:08.658015 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:08.658145 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:08.658920 (+ 775us) server_negotiation.cc:299] Negotiation successful
0926 16:47:08.659140 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":359,"threads_started":1}
I20260926 16:47:08.660435 24563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:08.637911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.195:41561)
0926 16:47:08.638862 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:08.638910 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:08.641040 (+ 2130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:08.642185 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:08.642220 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:08.643020 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:47:08.643898 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:08.643926 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:08.645526 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:08.645531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:08.647063 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:08.647077 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:08.648095 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:08.648101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:08.648370 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:08.649146 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:47:08.649199 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:47:08.651795 (+ 2596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:47:08.655824 (+ 4029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:08.655835 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:08.655853 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:08.656545 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:08.657140 (+ 595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:08.657145 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:08.657148 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:08.657412 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:08.658141 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:47:08.658148 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:47:08.658506 (+ 358us) client_negotiation.cc:769] Sending connection context
0926 16:47:08.658770 (+ 264us) client_negotiation.cc:240] Negotiation successful
0926 16:47:08.659230 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":417,"threads_started":1}
I20260926 16:47:08.662304 24560 heartbeater.cc:347] Connected to a master server at 127.9.238.254:46415
I20260926 16:47:08.662674 24560 heartbeater.cc:464] Registering TS with master...
I20260926 16:47:08.663655 24560 heartbeater.cc:511] Master 127.9.238.254:46415 requested a full tablet report, sending...
I20260926 16:47:08.665483 24103 ts_manager.cc:194] Registered new tserver with Master: fde78bbc9fed462a855a04094fde18da (127.9.238.195:38291)
I20260926 16:47:08.666373 24103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.195@KRBTEST.COM'} at 127.9.238.195:41561
I20260926 16:47:08.671088 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:47:08.690855 24562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:08.676969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45648 (local address 127.9.238.254:46415)
0926 16:47:08.677270 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:08.677280 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:08.677658 (+ 378us) server_negotiation.cc:407] Connection header received
0926 16:47:08.677894 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:08.677899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:08.677981 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:08.678093 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:08.679572 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:08.680458 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:08.681838 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:08.682237 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:08.682378 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:08.685148 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:08.685159 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:08.685162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:08.685196 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:08.688121 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:08.688778 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:08.688784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:08.688788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:08.688914 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:08.689294 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:08.689300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:08.689303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:08.689662 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:08.689934 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:08.690256 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 16:47:08.690518 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 16:47:08.696678 24103 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:47:08.704545 24103 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:47:08.752156 24577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:08.721589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:35439 (local address 127.0.0.1:57666)
0926 16:47:08.737108 (+ 15519us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:08.737141 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:08.737369 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:08.742667 (+ 5298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:08.742680 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:08.742789 (+ 109us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:47:08.743513 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:08.743535 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:08.746477 (+ 2942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:08.746486 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:08.748731 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:08.748744 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:08.750853 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:08.750860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:08.751164 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:08.751233 (+ 69us) client_negotiation.cc:769] Sending connection context
0926 16:47:08.751379 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 16:47:08.751646 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15275,"mutex_wait_us":37,"thread_start_us":21981,"threads_started":1}
I20260926 16:47:08.752527 24576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:08.721813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57666 (local address 127.9.238.194:35439)
0926 16:47:08.740851 (+ 19038us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:08.740870 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:08.740923 (+ 53us) server_negotiation.cc:407] Connection header received
0926 16:47:08.741102 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:08.741117 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:08.741612 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:08.741945 (+ 333us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:47:08.743984 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:08.746266 (+ 2282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:08.748927 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:08.750574 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:08.750766 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:08.751059 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:08.751490 (+ 431us) server_negotiation.cc:299] Negotiation successful
0926 16:47:08.752143 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18735,"thread_start_us":20413,"threads_started":1}
I20260926 16:47:08.754465 24575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:08.723054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:45243 (local address 127.0.0.1:38530)
0926 16:47:08.743954 (+ 20900us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:08.744003 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:08.744202 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:08.745421 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:08.745426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:08.745456 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:47:08.745828 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:08.745838 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:08.747814 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:08.747820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:08.750227 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:08.750240 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:08.753656 (+ 3416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:08.753662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:08.753838 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:08.753881 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 16:47:08.753994 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 16:47:08.754130 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20750,"thread_start_us":13582,"threads_started":1}
I20260926 16:47:08.754695 24578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:08.723828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38530 (local address 127.9.238.193:45243)
0926 16:47:08.742989 (+ 19161us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:08.743004 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:08.744429 (+ 1425us) server_negotiation.cc:407] Connection header received
0926 16:47:08.744557 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:08.744568 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:08.744892 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:08.745127 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:47:08.746012 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:08.747674 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:08.750456 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:08.753419 (+ 2963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:08.753589 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:08.753843 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:08.754052 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 16:47:08.754242 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18885,"thread_start_us":19186,"threads_started":1}
I20260926 16:47:08.756135 24574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:08.721135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41204 (local address 127.9.238.195:38291)
0926 16:47:08.746546 (+ 25411us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:08.746556 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:08.746590 (+ 34us) server_negotiation.cc:407] Connection header received
0926 16:47:08.746706 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:08.746714 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:08.746947 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:08.747161 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:47:08.747972 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:08.749611 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:08.752285 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:08.753834 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:08.755065 (+ 1231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:08.755349 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:08.755502 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 16:47:08.755722 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25190,"thread_start_us":25034,"threads_started":1}
I20260926 16:47:08.757452 24361 tablet_service.cc:1467] Processing CreateTablet for tablet f0b199db974b4bd5985dff6cdac645ca (DEFAULT_TABLE table=test-table [id=ac3e53ab80034343ac9cd6681f45398a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:47:08.758464 24227 tablet_service.cc:1467] Processing CreateTablet for tablet f0b199db974b4bd5985dff6cdac645ca (DEFAULT_TABLE table=test-table [id=ac3e53ab80034343ac9cd6681f45398a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:47:08.759159 24361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b199db974b4bd5985dff6cdac645ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:47:08.759312 24573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:08.720624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:38291 (local address 127.0.0.1:41204)
0926 16:47:08.722513 (+ 1889us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:08.722592 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:08.722879 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:08.747200 (+ 24321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:08.747206 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:08.747241 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:47:08.747732 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:08.747750 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:08.749790 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:08.749795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:08.752035 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:08.752056 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:08.755129 (+ 3073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:08.755135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:08.755307 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:08.755339 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 16:47:08.755441 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 16:47:08.755606 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1436,"mutex_wait_us":1,"thread_start_us":1360,"threads_started":1}
I20260926 16:47:08.760041 24227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b199db974b4bd5985dff6cdac645ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:47:08.763454 24495 tablet_service.cc:1467] Processing CreateTablet for tablet f0b199db974b4bd5985dff6cdac645ca (DEFAULT_TABLE table=test-table [id=ac3e53ab80034343ac9cd6681f45398a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:47:08.764760 24495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b199db974b4bd5985dff6cdac645ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:47:08.775435 24579 tablet_bootstrap.cc:492] T f0b199db974b4bd5985dff6cdac645ca P 0685cb7ee46548989fb7a170b111181a: Bootstrap starting.
I20260926 16:47:08.778276 24580 tablet_bootstrap.cc:492] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da: Bootstrap starting.
I20260926 16:47:08.780393 24579 tablet_bootstrap.cc:654] T f0b199db974b4bd5985dff6cdac645ca P 0685cb7ee46548989fb7a170b111181a: Neither blocks nor log segments found. Creating new log.
I20260926 16:47:08.782193 24579 log.cc:824] T f0b199db974b4bd5985dff6cdac645ca P 0685cb7ee46548989fb7a170b111181a: Log is configured to *not* fsync() on all Append() calls
I20260926 16:47:08.782658 24580 tablet_bootstrap.cc:654] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da: Neither blocks nor log segments found. Creating new log.
I20260926 16:47:08.784128 24580 log.cc:824] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da: Log is configured to *not* fsync() on all Append() calls
I20260926 16:47:08.784984 24582 tablet_bootstrap.cc:492] T f0b199db974b4bd5985dff6cdac645ca P 9a83f18a7e554e8ba00204ce1550a273: Bootstrap starting.
I20260926 16:47:08.786924 24579 tablet_bootstrap.cc:492] T f0b199db974b4bd5985dff6cdac645ca P 0685cb7ee46548989fb7a170b111181a: No bootstrap required, opened a new log
I20260926 16:47:08.787357 24579 ts_tablet_manager.cc:1397] T f0b199db974b4bd5985dff6cdac645ca P 0685cb7ee46548989fb7a170b111181a: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260926 16:47:08.788831 24582 tablet_bootstrap.cc:654] T f0b199db974b4bd5985dff6cdac645ca P 9a83f18a7e554e8ba00204ce1550a273: Neither blocks nor log segments found. Creating new log.
I20260926 16:47:08.789896 24582 log.cc:824] T f0b199db974b4bd5985dff6cdac645ca P 9a83f18a7e554e8ba00204ce1550a273: Log is configured to *not* fsync() on all Append() calls
I20260926 16:47:08.792989 24582 tablet_bootstrap.cc:492] T f0b199db974b4bd5985dff6cdac645ca P 9a83f18a7e554e8ba00204ce1550a273: No bootstrap required, opened a new log
I20260926 16:47:08.793316 24582 ts_tablet_manager.cc:1397] T f0b199db974b4bd5985dff6cdac645ca P 9a83f18a7e554e8ba00204ce1550a273: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260926 16:47:08.796222 24580 tablet_bootstrap.cc:492] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da: No bootstrap required, opened a new log
I20260926 16:47:08.796648 24580 ts_tablet_manager.cc:1397] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.000s
I20260926 16:47:08.799485 24579 raft_consensus.cc:348] T f0b199db974b4bd5985dff6cdac645ca P 0685cb7ee46548989fb7a170b111181a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde78bbc9fed462a855a04094fde18da" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 38291 } } peers { permanent_uuid: "9a83f18a7e554e8ba00204ce1550a273" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 45243 } } peers { permanent_uuid: "0685cb7ee46548989fb7a170b111181a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 35439 } }
I20260926 16:47:08.800204 24579 raft_consensus.cc:374] T f0b199db974b4bd5985dff6cdac645ca P 0685cb7ee46548989fb7a170b111181a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:47:08.800326 24579 raft_consensus.cc:729] T f0b199db974b4bd5985dff6cdac645ca P 0685cb7ee46548989fb7a170b111181a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0685cb7ee46548989fb7a170b111181a, State: Initialized, Role: FOLLOWER
I20260926 16:47:08.801276 24579 consensus_queue.cc:260] T f0b199db974b4bd5985dff6cdac645ca P 0685cb7ee46548989fb7a170b111181a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde78bbc9fed462a855a04094fde18da" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 38291 } } peers { permanent_uuid: "9a83f18a7e554e8ba00204ce1550a273" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 45243 } } peers { permanent_uuid: "0685cb7ee46548989fb7a170b111181a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 35439 } }
I20260926 16:47:08.801285 24582 raft_consensus.cc:348] T f0b199db974b4bd5985dff6cdac645ca P 9a83f18a7e554e8ba00204ce1550a273 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde78bbc9fed462a855a04094fde18da" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 38291 } } peers { permanent_uuid: "9a83f18a7e554e8ba00204ce1550a273" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 45243 } } peers { permanent_uuid: "0685cb7ee46548989fb7a170b111181a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 35439 } }
I20260926 16:47:08.801724 24582 raft_consensus.cc:374] T f0b199db974b4bd5985dff6cdac645ca P 9a83f18a7e554e8ba00204ce1550a273 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:47:08.801812 24582 raft_consensus.cc:729] T f0b199db974b4bd5985dff6cdac645ca P 9a83f18a7e554e8ba00204ce1550a273 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a83f18a7e554e8ba00204ce1550a273, State: Initialized, Role: FOLLOWER
I20260926 16:47:08.802500 24582 consensus_queue.cc:260] T f0b199db974b4bd5985dff6cdac645ca P 9a83f18a7e554e8ba00204ce1550a273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde78bbc9fed462a855a04094fde18da" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 38291 } } peers { permanent_uuid: "9a83f18a7e554e8ba00204ce1550a273" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 45243 } } peers { permanent_uuid: "0685cb7ee46548989fb7a170b111181a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 35439 } }
I20260926 16:47:08.803882 24579 ts_tablet_manager.cc:1428] T f0b199db974b4bd5985dff6cdac645ca P 0685cb7ee46548989fb7a170b111181a: Time spent starting tablet: real 0.016s user 0.009s sys 0.010s
I20260926 16:47:08.804531 24426 heartbeater.cc:503] Master 127.9.238.254:46415 was elected leader, sending a full tablet report...
I20260926 16:47:08.806417 24582 ts_tablet_manager.cc:1428] T f0b199db974b4bd5985dff6cdac645ca P 9a83f18a7e554e8ba00204ce1550a273: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260926 16:47:08.808698 24580 raft_consensus.cc:348] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde78bbc9fed462a855a04094fde18da" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 38291 } } peers { permanent_uuid: "9a83f18a7e554e8ba00204ce1550a273" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 45243 } } peers { permanent_uuid: "0685cb7ee46548989fb7a170b111181a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 35439 } }
I20260926 16:47:08.809479 24580 raft_consensus.cc:374] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:47:08.809625 24580 raft_consensus.cc:729] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde78bbc9fed462a855a04094fde18da, State: Initialized, Role: FOLLOWER
I20260926 16:47:08.810542 24580 consensus_queue.cc:260] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde78bbc9fed462a855a04094fde18da" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 38291 } } peers { permanent_uuid: "9a83f18a7e554e8ba00204ce1550a273" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 45243 } } peers { permanent_uuid: "0685cb7ee46548989fb7a170b111181a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 35439 } }
I20260926 16:47:08.811976 24560 heartbeater.cc:503] Master 127.9.238.254:46415 was elected leader, sending a full tablet report...
W20260926 16:47:08.812475 24293 tablet.cc:2369] T f0b199db974b4bd5985dff6cdac645ca P 9a83f18a7e554e8ba00204ce1550a273: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:47:08.813491 24580 ts_tablet_manager.cc:1428] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da: Time spent starting tablet: real 0.017s user 0.018s sys 0.000s
I20260926 16:47:08.868520 24587 raft_consensus.cc:482] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:47:08.868916 24587 raft_consensus.cc:504] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde78bbc9fed462a855a04094fde18da" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 38291 } } peers { permanent_uuid: "9a83f18a7e554e8ba00204ce1550a273" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 45243 } } peers { permanent_uuid: "0685cb7ee46548989fb7a170b111181a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 35439 } }
I20260926 16:47:08.871208 24587 leader_election.cc:288] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a83f18a7e554e8ba00204ce1550a273 (127.9.238.193:45243), 0685cb7ee46548989fb7a170b111181a (127.9.238.194:35439)
I20260926 16:47:08.878504 24563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:08.871528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:45243 (local address 127.9.238.195:55649)
0926 16:47:08.872063 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:08.872090 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:08.872287 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:08.872838 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:08.872842 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:08.872898 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:47:08.873177 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:08.873187 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:08.874471 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:08.874475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:08.876318 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:08.876331 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:08.877591 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:08.877598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:08.877801 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:08.877835 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 16:47:08.877953 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 16:47:08.878106 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":3}
I20260926 16:47:08.878703 24578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:08.871742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:55649 (local address 127.9.238.193:45243)
0926 16:47:08.872014 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:08.872022 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:08.872390 (+ 368us) server_negotiation.cc:407] Connection header received
0926 16:47:08.872498 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:08.872504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:08.872638 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:08.872774 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:47:08.873323 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:08.874318 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:08.876524 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:08.877402 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:08.877602 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:08.877690 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:08.878250 (+ 560us) server_negotiation.cc:299] Negotiation successful
0926 16:47:08.878425 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 16:47:08.879408 24576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:08.872094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:43213 (local address 127.9.238.194:35439)
0926 16:47:08.872320 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:08.872329 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:08.872988 (+ 659us) server_negotiation.cc:407] Connection header received
0926 16:47:08.873213 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:08.873217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:08.873308 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:08.873411 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:47:08.874032 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:08.874845 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:08.877068 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:08.878016 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:08.878205 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:08.878283 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:08.878929 (+ 646us) server_negotiation.cc:299] Negotiation successful
0926 16:47:08.879097 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 16:47:08.879930 24588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:08.871799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:35439 (local address 127.9.238.195:43213)
0926 16:47:08.872866 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:08.872890 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:08.873061 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:08.873443 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:08.873446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:08.873470 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:47:08.873872 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:08.873882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:08.875019 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:08.875026 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:08.876908 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:08.876922 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:08.878226 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:08.878233 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:08.878413 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:08.878451 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 16:47:08.879312 (+ 861us) client_negotiation.cc:240] Negotiation successful
0926 16:47:08.879512 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":61,"thread_start_us":548,"threads_started":1}
I20260926 16:47:08.880089 24247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b199db974b4bd5985dff6cdac645ca" candidate_uuid: "fde78bbc9fed462a855a04094fde18da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a83f18a7e554e8ba00204ce1550a273" is_pre_election: true
I20260926 16:47:08.880748 24247 raft_consensus.cc:2454] T f0b199db974b4bd5985dff6cdac645ca P 9a83f18a7e554e8ba00204ce1550a273 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde78bbc9fed462a855a04094fde18da in term 0.
I20260926 16:47:08.881186 24381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b199db974b4bd5985dff6cdac645ca" candidate_uuid: "fde78bbc9fed462a855a04094fde18da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0685cb7ee46548989fb7a170b111181a" is_pre_election: true
I20260926 16:47:08.881611 24451 leader_election.cc:302] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a83f18a7e554e8ba00204ce1550a273, fde78bbc9fed462a855a04094fde18da; no voters:
I20260926 16:47:08.881678 24381 raft_consensus.cc:2454] T f0b199db974b4bd5985dff6cdac645ca P 0685cb7ee46548989fb7a170b111181a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde78bbc9fed462a855a04094fde18da in term 0.
I20260926 16:47:08.882136 24587 raft_consensus.cc:2781] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:47:08.882334 24587 raft_consensus.cc:482] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:47:08.882475 24587 raft_consensus.cc:3037] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:47:08.885344 24587 raft_consensus.cc:504] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde78bbc9fed462a855a04094fde18da" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 38291 } } peers { permanent_uuid: "9a83f18a7e554e8ba00204ce1550a273" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 45243 } } peers { permanent_uuid: "0685cb7ee46548989fb7a170b111181a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 35439 } }
W20260926 16:47:08.885541 24561 tablet.cc:2369] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:47:08.886345 24587 leader_election.cc:288] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da [CANDIDATE]: Term 1 election: Requested vote from peers 9a83f18a7e554e8ba00204ce1550a273 (127.9.238.193:45243), 0685cb7ee46548989fb7a170b111181a (127.9.238.194:35439)
I20260926 16:47:08.886966 24247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b199db974b4bd5985dff6cdac645ca" candidate_uuid: "fde78bbc9fed462a855a04094fde18da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a83f18a7e554e8ba00204ce1550a273"
I20260926 16:47:08.886996 24381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b199db974b4bd5985dff6cdac645ca" candidate_uuid: "fde78bbc9fed462a855a04094fde18da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0685cb7ee46548989fb7a170b111181a"
I20260926 16:47:08.887241 24381 raft_consensus.cc:3037] T f0b199db974b4bd5985dff6cdac645ca P 0685cb7ee46548989fb7a170b111181a [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:47:08.887243 24247 raft_consensus.cc:3037] T f0b199db974b4bd5985dff6cdac645ca P 9a83f18a7e554e8ba00204ce1550a273 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:47:08.889829 24381 raft_consensus.cc:2454] T f0b199db974b4bd5985dff6cdac645ca P 0685cb7ee46548989fb7a170b111181a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde78bbc9fed462a855a04094fde18da in term 1.
I20260926 16:47:08.889942 24247 raft_consensus.cc:2454] T f0b199db974b4bd5985dff6cdac645ca P 9a83f18a7e554e8ba00204ce1550a273 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde78bbc9fed462a855a04094fde18da in term 1.
I20260926 16:47:08.890585 24450 leader_election.cc:302] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0685cb7ee46548989fb7a170b111181a, fde78bbc9fed462a855a04094fde18da; no voters:
I20260926 16:47:08.891023 24587 raft_consensus.cc:2781] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:47:08.891685 24587 raft_consensus.cc:686] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da [term 1 LEADER]: Becoming Leader. State: Replica: fde78bbc9fed462a855a04094fde18da, State: Running, Role: LEADER
I20260926 16:47:08.892457 24587 consensus_queue.cc:237] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde78bbc9fed462a855a04094fde18da" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 38291 } } peers { permanent_uuid: "9a83f18a7e554e8ba00204ce1550a273" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 45243 } } peers { permanent_uuid: "0685cb7ee46548989fb7a170b111181a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 35439 } }
I20260926 16:47:08.898613 24102 catalog_manager.cc:5495] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da reported cstate change: term changed from 0 to 1, leader changed from <none> to fde78bbc9fed462a855a04094fde18da (127.9.238.195). New cstate: current_term: 1 leader_uuid: "fde78bbc9fed462a855a04094fde18da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde78bbc9fed462a855a04094fde18da" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 38291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a83f18a7e554e8ba00204ce1550a273" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 45243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0685cb7ee46548989fb7a170b111181a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 35439 } health_report { overall_health: UNKNOWN } } }
I20260926 16:47:08.932624 24574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:08.925903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41216 (local address 127.9.238.195:38291)
0926 16:47:08.926127 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:08.926132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:08.926153 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:47:08.926281 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:08.926285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:08.926401 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:08.926580 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:47:08.926938 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:08.927927 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:08.929559 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:08.930021 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:08.930137 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:08.930899 (+ 762us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:47:08.931817 (+ 918us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:47:08.932015 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:08.932186 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 16:47:08.932349 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 16:47:08.943521 24381 raft_consensus.cc:1261] T f0b199db974b4bd5985dff6cdac645ca P 0685cb7ee46548989fb7a170b111181a [term 1 FOLLOWER]: Refusing update from remote peer fde78bbc9fed462a855a04094fde18da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:47:08.943650 24247 raft_consensus.cc:1261] T f0b199db974b4bd5985dff6cdac645ca P 9a83f18a7e554e8ba00204ce1550a273 [term 1 FOLLOWER]: Refusing update from remote peer fde78bbc9fed462a855a04094fde18da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:47:08.944958 24589 consensus_queue.cc:1035] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a83f18a7e554e8ba00204ce1550a273" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 45243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:47:08.945370 24587 consensus_queue.cc:1035] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0685cb7ee46548989fb7a170b111181a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 35439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:47:08.959817 24592 mvcc.cc:205] Tried to move back new op lower bound from 7333647273739042816 to 7333647273548771328. Current Snapshot: MvccSnapshot[applied={T|T < 7333647273739042816}]
I20260926 16:47:08.963057 24594 mvcc.cc:205] Tried to move back new op lower bound from 7333647273739042816 to 7333647273548771328. Current Snapshot: MvccSnapshot[applied={T|T < 7333647273739042816}]
I20260926 16:47:08.980387 24102 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:47:08.980806 24102 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:47:08.985157 24102 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad: Sending DeleteTablet for 3 replicas of tablet f0b199db974b4bd5985dff6cdac645ca
I20260926 16:47:08.986740 24361 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b199db974b4bd5985dff6cdac645ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:47:08 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:57666
I20260926 16:47:08.987066 24227 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b199db974b4bd5985dff6cdac645ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:47:08 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:38530
I20260926 16:47:08.987063 24495 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b199db974b4bd5985dff6cdac645ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:47:08 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:41204
I20260926 16:47:08.988520 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 24164
I20260926 16:47:09.001416 24602 tablet_replica.cc:330] T f0b199db974b4bd5985dff6cdac645ca P 0685cb7ee46548989fb7a170b111181a: stopping tablet replica
I20260926 16:47:09.002131 24603 tablet_replica.cc:330] T f0b199db974b4bd5985dff6cdac645ca P 9a83f18a7e554e8ba00204ce1550a273: stopping tablet replica
I20260926 16:47:09.002988 24603 raft_consensus.cc:2229] T f0b199db974b4bd5985dff6cdac645ca P 9a83f18a7e554e8ba00204ce1550a273 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:47:09.003623 24602 raft_consensus.cc:2229] T f0b199db974b4bd5985dff6cdac645ca P 0685cb7ee46548989fb7a170b111181a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:47:09.004426 24603 raft_consensus.cc:2258] T f0b199db974b4bd5985dff6cdac645ca P 9a83f18a7e554e8ba00204ce1550a273 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:47:09.005019 24602 raft_consensus.cc:2258] T f0b199db974b4bd5985dff6cdac645ca P 0685cb7ee46548989fb7a170b111181a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:47:09.006232 24604 tablet_replica.cc:330] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da: stopping tablet replica
I20260926 16:47:09.006752 24604 raft_consensus.cc:2229] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:47:09.007534 24604 raft_consensus.cc:2258] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:47:09.008033 24602 ts_tablet_manager.cc:1905] T f0b199db974b4bd5985dff6cdac645ca P 0685cb7ee46548989fb7a170b111181a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:47:09.013661 24604 ts_tablet_manager.cc:1905] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:47:09.016888 24602 ts_tablet_manager.cc:1918] T f0b199db974b4bd5985dff6cdac645ca P 0685cb7ee46548989fb7a170b111181a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:47:09.017102 24602 log.cc:1196] T f0b199db974b4bd5985dff6cdac645ca P 0685cb7ee46548989fb7a170b111181a: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/wal/wals/f0b199db974b4bd5985dff6cdac645ca
I20260926 16:47:09.017737 24602 ts_tablet_manager.cc:1939] T f0b199db974b4bd5985dff6cdac645ca P 0685cb7ee46548989fb7a170b111181a: Deleting consensus metadata
Sep 26 16:47:09 dist-test-slave-wt2j krb5kdc[24058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.193@KRBTEST.COM
I20260926 16:47:09.019968 24603 ts_tablet_manager.cc:1905] T f0b199db974b4bd5985dff6cdac645ca P 9a83f18a7e554e8ba00204ce1550a273: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:47:09.020323 24089 catalog_manager.cc:4841] TS 0685cb7ee46548989fb7a170b111181a (127.9.238.194:35439): tablet f0b199db974b4bd5985dff6cdac645ca (table test-table [id=ac3e53ab80034343ac9cd6681f45398a]) successfully deleted
I20260926 16:47:09.021751 24604 ts_tablet_manager.cc:1918] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:47:09.021945 24604 log.cc:1196] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/wal/wals/f0b199db974b4bd5985dff6cdac645ca
I20260926 16:47:09.022574 24604 ts_tablet_manager.cc:1939] T f0b199db974b4bd5985dff6cdac645ca P fde78bbc9fed462a855a04094fde18da: Deleting consensus metadata
I20260926 16:47:09.024649 24090 catalog_manager.cc:4841] TS fde78bbc9fed462a855a04094fde18da (127.9.238.195:38291): tablet f0b199db974b4bd5985dff6cdac645ca (table test-table [id=ac3e53ab80034343ac9cd6681f45398a]) successfully deleted
I20260926 16:47:09.027627 24603 ts_tablet_manager.cc:1918] T f0b199db974b4bd5985dff6cdac645ca P 9a83f18a7e554e8ba00204ce1550a273: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:47:09.027859 24603 log.cc:1196] T f0b199db974b4bd5985dff6cdac645ca P 9a83f18a7e554e8ba00204ce1550a273: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/wal/wals/f0b199db974b4bd5985dff6cdac645ca
I20260926 16:47:09.028486 24603 ts_tablet_manager.cc:1939] T f0b199db974b4bd5985dff6cdac645ca P 9a83f18a7e554e8ba00204ce1550a273: Deleting consensus metadata
I20260926 16:47:09.030668 24090 catalog_manager.cc:4841] TS 9a83f18a7e554e8ba00204ce1550a273 (127.9.238.193:45243): tablet f0b199db974b4bd5985dff6cdac645ca (table test-table [id=ac3e53ab80034343ac9cd6681f45398a]) successfully deleted
I20260926 16:47:09.032627 24578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:08.989965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38534 (local address 127.9.238.193:45243)
0926 16:47:08.990154 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:08.990159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:08.999028 (+ 8869us) server_negotiation.cc:407] Connection header received
0926 16:47:08.999285 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:08.999295 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:08.999438 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:08.999627 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:09.008144 (+ 8517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:09.009315 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:09.012417 (+ 3102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:09.013225 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:09.013350 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:09.021707 (+ 8357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:09.021740 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:09.021766 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:09.021861 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:09.028058 (+ 6197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:09.029355 (+ 1297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:09.029370 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:09.029374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:09.029647 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:09.030391 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:09.030407 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:09.030411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:09.031061 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:09.031459 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:09.031866 (+ 407us) server_negotiation.cc:299] Negotiation successful
0926 16:47:09.032173 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 16:47:09.033638 24287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:47:09.182057 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 24164
I20260926 16:47:09.202165 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 24298
Sep 26 16:47:09 dist-test-slave-wt2j krb5kdc[24058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.194@KRBTEST.COM
I20260926 16:47:09.224789 24576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:09.203321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57674 (local address 127.9.238.194:35439)
0926 16:47:09.203634 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:09.203639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:09.203656 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:47:09.203709 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:09.203712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:09.203782 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:09.203890 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:09.205154 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:09.205929 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:09.207195 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:09.207653 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:09.207774 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:09.218057 (+ 10283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:09.218077 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:09.218096 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:09.218142 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:09.221597 (+ 3455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:09.222429 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:09.222437 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:09.222440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:09.222591 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:09.222994 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:09.223002 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:09.223005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:09.223435 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:09.223807 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:09.224151 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 16:47:09.224394 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260926 16:47:09.225833 24421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:47:09.318642 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 24298
I20260926 16:47:09.346912 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 24432
Sep 26 16:47:09 dist-test-slave-wt2j krb5kdc[24058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.195@KRBTEST.COM
I20260926 16:47:09.369199 24574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:09.348004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41218 (local address 127.9.238.195:38291)
0926 16:47:09.348227 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:09.348231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:09.348310 (+ 79us) server_negotiation.cc:407] Connection header received
0926 16:47:09.348565 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:09.348569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:09.348660 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:09.348758 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:09.350708 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:09.351826 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:09.354022 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:09.354528 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:09.354663 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:09.361455 (+ 6792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:09.361482 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:09.361500 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:09.361555 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:09.365241 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:09.366448 (+ 1207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:09.366455 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:09.366457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:09.366669 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:09.367210 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:09.367215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:09.367218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:09.367834 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:09.368121 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:09.368612 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 16:47:09.368818 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 16:47:09.369961 24555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:47:09.508559 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 24432
I20260926 16:47:09.529788 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 24071
I20260926 16:47:09.531281 24133 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:47:09.649554 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 24071
I20260926 16:47:09.669910 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:46415
--webserver_interface=127.9.238.254
--webserver_port=34423
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:36335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:46415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:47:10.090258 24610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:47:10.090739 24610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:47:10.090924 24610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:47:10.099661 24610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:47:10.099869 24610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:47:10.099929 24610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:47:10.100004 24610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:47:10.100059 24610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:47:10.111966 24610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:36335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:46415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:46415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=34423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:47:10.113709 24610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:47:10.115332 24610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:47:10.146363 24619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:10.146477 24616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:10.146411 24617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:47:10.147745 24610 server_base.cc:1034] running on GCE node
I20260926 16:47:10.148803 24610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:47:10.150822 24610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:47:10.152155 24610 hybrid_clock.cc:648] HybridClock initialized: now 1790441230152080 us; error 67 us; skew 500 ppm
I20260926 16:47:10.152663 24610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:47:10 dist-test-slave-wt2j krb5kdc[24058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441230, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:47:10.162758 24610 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:47:10.165064 24610 webserver.cc:460] Webserver started at http://127.9.238.254:34423/ using document root <none> and password file <none>
I20260926 16:47:10.165969 24610 fs_manager.cc:359] Metadata directory not provided
I20260926 16:47:10.166150 24610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:47:10.173465 24610 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260926 16:47:10.177435 24626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:10.179476 24610 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:47:10.179790 24610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "2a81ad67dca84ffabddd19f5d60e3cad"
format_stamp: "Formatted at 2026-09-26 16:47:06 on dist-test-slave-wt2j"
server_key: "c419107dd2a6a00cced3eb5c7ff9c412"
server_key_iv: "049fca501495e8e60d3582a1b40ec1c4"
server_key_version: "encryptionkey@0"
I20260926 16:47:10.181044 24610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:47:10.201869 24610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:47:10.207314 24610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:47:10.207732 24610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:47:10.230674 24610 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:46415
I20260926 16:47:10.230758 24677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:46415 every 8 connection(s)
I20260926 16:47:10.232555 24610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:47:10.241878 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 24610
I20260926 16:47:10.243106 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:45243
--local_ip_for_outbound_sockets=127.9.238.193
--tserver_master_addrs=127.9.238.254:46415
--webserver_port=39237
--webserver_interface=127.9.238.193
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:36335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:47:10.243383 24678 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:47:10.250320 24678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad: Bootstrap starting.
I20260926 16:47:10.278787 24678 log.cc:824] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad: Log is configured to *not* fsync() on all Append() calls
I20260926 16:47:10.295785 24678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:47:10.296535 24678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad: Bootstrap complete.
I20260926 16:47:10.308853 24678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a81ad67dca84ffabddd19f5d60e3cad" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46415 } }
I20260926 16:47:10.310070 24678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a81ad67dca84ffabddd19f5d60e3cad, State: Initialized, Role: FOLLOWER
I20260926 16:47:10.310976 24678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a81ad67dca84ffabddd19f5d60e3cad" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46415 } }
I20260926 16:47:10.311331 24678 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:47:10.311508 24678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:47:10.311858 24678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [term 1 FOLLOWER]: Advancing to term 2
I20260926 16:47:10.317984 24678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a81ad67dca84ffabddd19f5d60e3cad" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46415 } }
I20260926 16:47:10.318859 24678 leader_election.cc:302] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a81ad67dca84ffabddd19f5d60e3cad; no voters:
I20260926 16:47:10.319765 24678 leader_election.cc:288] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 16:47:10.319967 24682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [term 2 FOLLOWER]: Leader election won for term 2
I20260926 16:47:10.321545 24682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [term 2 LEADER]: Becoming Leader. State: Replica: 2a81ad67dca84ffabddd19f5d60e3cad, State: Running, Role: LEADER
I20260926 16:47:10.322300 24682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a81ad67dca84ffabddd19f5d60e3cad" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46415 } }
I20260926 16:47:10.322953 24678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:47:10.325783 24683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2a81ad67dca84ffabddd19f5d60e3cad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a81ad67dca84ffabddd19f5d60e3cad" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46415 } } }
I20260926 16:47:10.326408 24683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [sys.catalog]: This master's current role is: LEADER
I20260926 16:47:10.327194 24684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a81ad67dca84ffabddd19f5d60e3cad. Latest consensus state: current_term: 2 leader_uuid: "2a81ad67dca84ffabddd19f5d60e3cad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a81ad67dca84ffabddd19f5d60e3cad" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46415 } } }
I20260926 16:47:10.327804 24684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [sys.catalog]: This master's current role is: LEADER
I20260926 16:47:10.329836 24687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:47:10.342011 24687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:47:10.343959 24687 catalog_manager.cc:1259] Loaded cluster ID: 58ed3eb0b84b4b2b804dc006c690be83
I20260926 16:47:10.344583 24687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:47:10.348739 24687 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 16:47:10.361480 24699 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 16:47:10.368381 24687 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 16:47:10.379570 24687 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:47:10.381438 24687 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad: Loaded TSK: 0
I20260926 16:47:10.383046 24687 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 16:47:10.644759 24680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:47:10.645051 24680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:47:10.645179 24680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:47:10.652784 24680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:47:10.652931 24680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:47:10.653107 24680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:47:10.663367 24680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:36335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:45243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=39237
--tserver_master_addrs=127.9.238.254:46415
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:47:10.664841 24680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:47:10.666246 24680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:47:10.684271 24705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:10.684193 24706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:10.684231 24708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:47:10.685693 24680 server_base.cc:1034] running on GCE node
I20260926 16:47:10.686504 24680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:47:10.687700 24680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:47:10.689028 24680 hybrid_clock.cc:648] HybridClock initialized: now 1790441230688968 us; error 53 us; skew 500 ppm
I20260926 16:47:10.689378 24680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:47:10 dist-test-slave-wt2j krb5kdc[24058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441230, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:47:10.693292 24680 init.cc:377] Logged in from keytab as kudu/127.9.238.193@KRBTEST.COM (short username kudu)
I20260926 16:47:10.695050 24680 webserver.cc:460] Webserver started at http://127.9.238.193:39237/ using document root <none> and password file <none>
I20260926 16:47:10.695873 24680 fs_manager.cc:359] Metadata directory not provided
I20260926 16:47:10.696036 24680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:47:10.702541 24680 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 16:47:10.706527 24715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:10.708542 24680 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 16:47:10.708782 24680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "9a83f18a7e554e8ba00204ce1550a273"
format_stamp: "Formatted at 2026-09-26 16:47:07 on dist-test-slave-wt2j"
server_key: "6d01d3f70319ed0ddcf6826a5ce62ace"
server_key_iv: "95742a9c60913135bbf7407dd93a1092"
server_key_version: "encryptionkey@0"
I20260926 16:47:10.709909 24680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:47:10.728565 24680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:47:10.733431 24680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:47:10.733716 24680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:47:10.734814 24680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:47:10.736442 24680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:47:10.736536 24680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:10.736671 24680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:47:10.736776 24680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:10.778844 24680 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:45243
I20260926 16:47:10.779033 24827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:45243 every 8 connection(s)
I20260926 16:47:10.780532 24680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:47:10.786763 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 24680
I20260926 16:47:10.787518 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:35439
--local_ip_for_outbound_sockets=127.9.238.194
--tserver_master_addrs=127.9.238.254:46415
--webserver_port=40805
--webserver_interface=127.9.238.194
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:36335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:47:10 dist-test-slave-wt2j krb5kdc[24058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441230, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:47:10.813045 24830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:10.784309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:49463 (local address 127.9.238.254:46415)
0926 16:47:10.784997 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:10.785011 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:10.785729 (+ 718us) server_negotiation.cc:407] Connection header received
0926 16:47:10.788399 (+ 2670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:10.788429 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:10.789122 (+ 693us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:10.789729 (+ 607us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:10.791391 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:10.793971 (+ 2580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:10.796012 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:10.797440 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:10.797748 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:10.803026 (+ 5278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:10.803038 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:10.803062 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:10.803129 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:10.806781 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:10.808106 (+ 1325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:10.808116 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:10.808124 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:10.808376 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:10.808968 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:10.808972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:10.808975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:10.809604 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:10.809905 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:10.811116 (+ 1211us) server_negotiation.cc:299] Negotiation successful
0926 16:47:10.811749 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":321,"threads_started":1}
I20260926 16:47:10.813406 24831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:10.784514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.193:49463)
0926 16:47:10.785498 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:10.785553 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:10.787617 (+ 2064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:10.789860 (+ 2243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:10.789870 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:10.790417 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:47:10.791053 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:10.791071 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:10.794237 (+ 3166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:10.794241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:10.795705 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:10.795718 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:10.797803 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:10.797809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:10.798118 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:10.799070 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:47:10.799111 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:47:10.802567 (+ 3456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:47:10.807049 (+ 4482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:10.807063 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:10.807091 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:10.807888 (+ 797us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:10.808528 (+ 640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:10.808537 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:10.808543 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:10.808831 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:10.809782 (+ 951us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:47:10.809794 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:47:10.810384 (+ 590us) client_negotiation.cc:769] Sending connection context
0926 16:47:10.810781 (+ 397us) client_negotiation.cc:240] Negotiation successful
0926 16:47:10.811485 (+ 704us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":470,"threads_started":1}
I20260926 16:47:10.817489 24828 heartbeater.cc:347] Connected to a master server at 127.9.238.254:46415
I20260926 16:47:10.818008 24828 heartbeater.cc:464] Registering TS with master...
I20260926 16:47:10.819134 24828 heartbeater.cc:511] Master 127.9.238.254:46415 requested a full tablet report, sending...
I20260926 16:47:10.822571 24643 ts_manager.cc:194] Registered new tserver with Master: 9a83f18a7e554e8ba00204ce1550a273 (127.9.238.193:45243)
I20260926 16:47:10.825182 24643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.193@KRBTEST.COM'} at 127.9.238.193:49463
W20260926 16:47:11.138850 24832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:47:11.139241 24832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:47:11.139424 24832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:47:11.147183 24832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:47:11.147385 24832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:47:11.147686 24832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:47:11.158053 24832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:36335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:35439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=40805
--tserver_master_addrs=127.9.238.254:46415
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:47:11.159531 24832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:47:11.160897 24832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:47:11.176414 24838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:11.176466 24840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:11.176466 24837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:47:11.182858 24832 server_base.cc:1034] running on GCE node
I20260926 16:47:11.183792 24832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:47:11.185070 24832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:47:11.186555 24832 hybrid_clock.cc:648] HybridClock initialized: now 1790441231186455 us; error 81 us; skew 500 ppm
I20260926 16:47:11.186960 24832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:47:11 dist-test-slave-wt2j krb5kdc[24058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441231, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:47:11.191355 24832 init.cc:377] Logged in from keytab as kudu/127.9.238.194@KRBTEST.COM (short username kudu)
I20260926 16:47:11.193604 24832 webserver.cc:460] Webserver started at http://127.9.238.194:40805/ using document root <none> and password file <none>
I20260926 16:47:11.194438 24832 fs_manager.cc:359] Metadata directory not provided
I20260926 16:47:11.194620 24832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:47:11.201048 24832 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:47:11.204877 24847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:11.206853 24832 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:47:11.207100 24832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "0685cb7ee46548989fb7a170b111181a"
format_stamp: "Formatted at 2026-09-26 16:47:07 on dist-test-slave-wt2j"
server_key: "1691e2ee85552ffcd7a026cacd9a0501"
server_key_iv: "ecb1b09a3cb01e428f4cde3b2287f1f0"
server_key_version: "encryptionkey@0"
I20260926 16:47:11.208289 24832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:47:11.231150 24832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:47:11.236773 24832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:47:11.237214 24832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:47:11.238503 24832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:47:11.240327 24832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:47:11.240481 24832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:11.240665 24832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:47:11.240765 24832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:11.285244 24832 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:35439
I20260926 16:47:11.285418 24959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:35439 every 8 connection(s)
I20260926 16:47:11.287035 24832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:47:11.292385 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 24832
I20260926 16:47:11.293219 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:38291
--local_ip_for_outbound_sockets=127.9.238.195
--tserver_master_addrs=127.9.238.254:46415
--webserver_port=36199
--webserver_interface=127.9.238.195
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:36335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:47:11 dist-test-slave-wt2j krb5kdc[24058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441231, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:47:11.317446 24830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:11.290692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:36443 (local address 127.9.238.254:46415)
0926 16:47:11.290983 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:11.290988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:11.292844 (+ 1856us) server_negotiation.cc:407] Connection header received
0926 16:47:11.296149 (+ 3305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:11.296155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:11.296322 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:11.296453 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:11.299019 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:11.300036 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:11.302367 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:11.303229 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:11.303421 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:11.308128 (+ 4707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:11.308139 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:11.308148 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:11.308215 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:11.312619 (+ 4404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:11.313902 (+ 1283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:11.313912 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:11.313918 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:11.314142 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:11.314868 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:11.314876 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:11.314882 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:11.315370 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:11.315592 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:11.316530 (+ 938us) server_negotiation.cc:299] Negotiation successful
0926 16:47:11.316900 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260926 16:47:11.318126 24962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:11.291309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.194:36443)
0926 16:47:11.292583 (+ 1274us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:11.292650 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:11.295670 (+ 3020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:11.296829 (+ 1159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:11.296844 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:11.297742 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:47:11.298681 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:11.298712 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:11.300266 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:11.300274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:11.302138 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:11.302152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:11.303620 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:11.303632 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:11.303868 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:11.304834 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:47:11.304863 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:47:11.307771 (+ 2908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:47:11.312913 (+ 5142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:11.312922 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:11.312938 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:11.313690 (+ 752us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:11.314318 (+ 628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:11.314331 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:11.314334 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:11.314702 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:11.315628 (+ 926us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:47:11.315637 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:47:11.316060 (+ 423us) client_negotiation.cc:769] Sending connection context
0926 16:47:11.316366 (+ 306us) client_negotiation.cc:240] Negotiation successful
0926 16:47:11.316804 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"thread_start_us":511,"threads_started":1}
I20260926 16:47:11.320075 24960 heartbeater.cc:347] Connected to a master server at 127.9.238.254:46415
I20260926 16:47:11.320416 24960 heartbeater.cc:464] Registering TS with master...
I20260926 16:47:11.321192 24960 heartbeater.cc:511] Master 127.9.238.254:46415 requested a full tablet report, sending...
I20260926 16:47:11.323119 24643 ts_manager.cc:194] Registered new tserver with Master: 0685cb7ee46548989fb7a170b111181a (127.9.238.194:35439)
I20260926 16:47:11.324321 24643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.194@KRBTEST.COM'} at 127.9.238.194:36443
W20260926 16:47:11.687207 24963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:47:11.687505 24963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:47:11.687706 24963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:47:11.695773 24963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:47:11.695904 24963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:47:11.696102 24963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:47:11.706413 24963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:36335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:38291
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=36199
--tserver_master_addrs=127.9.238.254:46415
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:47:11.707883 24963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:47:11.709282 24963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:47:11.726364 24971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:11.726392 24968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:11.726493 24969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:47:11.727757 24963 server_base.cc:1034] running on GCE node
I20260926 16:47:11.728593 24963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:47:11.729740 24963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:47:11.731093 24963 hybrid_clock.cc:648] HybridClock initialized: now 1790441231731025 us; error 61 us; skew 500 ppm
I20260926 16:47:11.731617 24963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:47:11 dist-test-slave-wt2j krb5kdc[24058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441231, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:47:11.735778 24963 init.cc:377] Logged in from keytab as kudu/127.9.238.195@KRBTEST.COM (short username kudu)
I20260926 16:47:11.737722 24963 webserver.cc:460] Webserver started at http://127.9.238.195:36199/ using document root <none> and password file <none>
I20260926 16:47:11.738732 24963 fs_manager.cc:359] Metadata directory not provided
I20260926 16:47:11.738864 24963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:47:11.745455 24963 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:47:11.749240 24978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:11.751111 24963 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 16:47:11.751341 24963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "fde78bbc9fed462a855a04094fde18da"
format_stamp: "Formatted at 2026-09-26 16:47:08 on dist-test-slave-wt2j"
server_key: "9ac1b971a5e4869a36e58713887f9265"
server_key_iv: "f3887687ab9409bb9fddff75e8066089"
server_key_version: "encryptionkey@0"
I20260926 16:47:11.752666 24963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:47:11.767882 24963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:47:11.773017 24963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:47:11.773324 24963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:47:11.774551 24963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:47:11.776162 24963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:47:11.776290 24963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:11.776433 24963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:47:11.776531 24963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:11.817554 24963 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:38291
I20260926 16:47:11.817611 25090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:38291 every 8 connection(s)
I20260926 16:47:11.819298 24963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:47:11.829845 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 24963
I20260926 16:47:11.829994 24828 heartbeater.cc:503] Master 127.9.238.254:46415 was elected leader, sending a full tablet report...
Sep 26 16:47:11 dist-test-slave-wt2j krb5kdc[24058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441231, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:47:11.847074 25093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:11.822715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:50869 (local address 127.9.238.254:46415)
0926 16:47:11.823351 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:11.823356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:11.824642 (+ 1286us) server_negotiation.cc:407] Connection header received
0926 16:47:11.827033 (+ 2391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:11.827039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:11.827161 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:11.827307 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:11.829722 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:11.830958 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:11.834009 (+ 3051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:11.834623 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:11.834734 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:11.839326 (+ 4592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:11.839333 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:11.839340 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:11.839400 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:11.843046 (+ 3646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:11.844093 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:11.844099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:11.844103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:11.844306 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:11.844912 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:11.844919 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:11.844921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:11.845390 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:11.845607 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:11.846366 (+ 759us) server_negotiation.cc:299] Negotiation successful
0926 16:47:11.846675 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":351,"threads_started":1}
I20260926 16:47:11.848259 25094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:11.823346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.195:50869)
0926 16:47:11.824371 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:11.824470 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:11.826628 (+ 2158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:11.827683 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:11.827694 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:11.828408 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:47:11.829194 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:11.829217 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:11.831271 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:11.831279 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:11.833787 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:11.833815 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:11.834911 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:11.834920 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:11.835307 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:11.836256 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:47:11.836281 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:47:11.839035 (+ 2754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:47:11.843236 (+ 4201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:11.843245 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:11.843260 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:11.843927 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:11.844463 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:11.844467 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:11.844470 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:11.844704 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:11.845561 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:47:11.845570 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:47:11.845954 (+ 384us) client_negotiation.cc:769] Sending connection context
0926 16:47:11.846246 (+ 292us) client_negotiation.cc:240] Negotiation successful
0926 16:47:11.846825 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":383,"threads_started":1}
I20260926 16:47:11.850044 25091 heartbeater.cc:347] Connected to a master server at 127.9.238.254:46415
I20260926 16:47:11.850327 25095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:11.831183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47568 (local address 127.9.238.254:46415)
0926 16:47:11.832913 (+ 1730us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:11.832920 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:11.833519 (+ 599us) server_negotiation.cc:407] Connection header received
0926 16:47:11.833774 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:11.833779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:11.833860 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:11.834009 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:11.836209 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:11.837041 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:11.838763 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:11.839375 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:11.839632 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:11.842633 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:11.842642 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:11.842650 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:11.842702 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:11.846166 (+ 3464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:11.847071 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:11.847077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:11.847081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:11.847433 (+ 352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:11.848081 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:11.848086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:11.848088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:11.848609 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:11.848950 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:11.849367 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 16:47:11.849673 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1562,"thread_start_us":1672,"threads_started":1}
I20260926 16:47:11.850426 25091 heartbeater.cc:464] Registering TS with master...
I20260926 16:47:11.851511 25091 heartbeater.cc:511] Master 127.9.238.254:46415 requested a full tablet report, sending...
I20260926 16:47:11.853569 24643 ts_manager.cc:194] Registered new tserver with Master: fde78bbc9fed462a855a04094fde18da (127.9.238.195:38291)
I20260926 16:47:11.854508 24643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.195@KRBTEST.COM'} at 127.9.238.195:50869
I20260926 16:47:11.868193 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:47:11.878435 25095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:11.869409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47572 (local address 127.9.238.254:46415)
0926 16:47:11.869700 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:11.869711 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:11.870457 (+ 746us) server_negotiation.cc:407] Connection header received
0926 16:47:11.870612 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:11.870622 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:11.870877 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:11.871085 (+ 208us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:47:11.871862 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:11.872797 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:11.875126 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:11.875814 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:11.875967 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:11.876727 (+ 760us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:47:11.877490 (+ 763us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:47:11.877650 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:11.877860 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 16:47:11.877972 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 16:47:11.879913 24643 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:47572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:47:11.889048 24643 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:47:11.930624 25104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:11.905713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:35439 (local address 127.0.0.1:51990)
0926 16:47:11.919881 (+ 14168us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:11.919906 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:11.920131 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:11.921216 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:11.921221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:11.921250 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:47:11.921697 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:11.921714 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:11.924331 (+ 2617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:11.924339 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:11.927136 (+ 2797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:11.927158 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:11.929255 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:11.929263 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:11.929525 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:11.929638 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 16:47:11.929778 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 16:47:11.930066 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14011,"mutex_wait_us":113,"thread_start_us":13851,"threads_started":1}
I20260926 16:47:11.931083 25105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:11.906164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51990 (local address 127.9.238.194:35439)
0926 16:47:11.920399 (+ 14235us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:11.920410 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:11.920447 (+ 37us) server_negotiation.cc:407] Connection header received
0926 16:47:11.920574 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:11.920582 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:11.920955 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:11.921238 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:47:11.921982 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:11.924119 (+ 2137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:11.927410 (+ 3291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:11.928955 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:11.930130 (+ 1175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:11.930372 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:11.930455 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 16:47:11.930697 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14005,"thread_start_us":15291,"threads_started":1}
I20260926 16:47:11.931506 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:11.905598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35834 (local address 127.9.238.193:45243)
0926 16:47:11.918123 (+ 12525us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:11.918136 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:11.919415 (+ 1279us) server_negotiation.cc:407] Connection header received
0926 16:47:11.919583 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:11.919591 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:11.919867 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:11.921140 (+ 1273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:47:11.921523 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:11.923783 (+ 2260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:11.927521 (+ 3738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:11.929288 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:11.929441 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:11.929779 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:11.930537 (+ 758us) server_negotiation.cc:299] Negotiation successful
0926 16:47:11.930903 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12228,"thread_start_us":12000,"threads_started":1}
I20260926 16:47:11.931859 25102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:11.905331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:45243 (local address 127.0.0.1:35834)
0926 16:47:11.918740 (+ 13409us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:11.918794 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:11.919162 (+ 368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:11.920162 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:11.920174 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:11.920225 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:47:11.920846 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:11.920864 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:11.924008 (+ 3144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:11.924017 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:11.927186 (+ 3169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:11.927203 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:11.929971 (+ 2768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:11.929978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:11.930272 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:11.930323 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 16:47:11.931219 (+ 896us) client_negotiation.cc:240] Negotiation successful
0926 16:47:11.931402 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13094,"mutex_wait_us":114,"thread_start_us":12881,"threads_started":1}
I20260926 16:47:11.935806 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:11.920476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:38291 (local address 127.0.0.1:35486)
0926 16:47:11.925141 (+ 4665us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:11.925164 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:11.925323 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:11.926528 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:11.926534 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:11.926573 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:47:11.927166 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:11.927175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:11.929437 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:11.929443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:11.932546 (+ 3103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:11.932564 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:11.934644 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:11.934652 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:11.934923 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:11.934975 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 16:47:11.935115 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 16:47:11.935280 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4465,"mutex_wait_us":63,"thread_start_us":4253,"threads_started":1}
I20260926 16:47:11.936471 25106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:11.919706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35486 (local address 127.9.238.195:38291)
0926 16:47:11.924596 (+ 4890us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:11.924609 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:11.925477 (+ 868us) server_negotiation.cc:407] Connection header received
0926 16:47:11.925606 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:11.925614 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:11.925907 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:11.926517 (+ 610us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:47:11.927454 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:11.929250 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:11.932791 (+ 3541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:11.934346 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:11.934493 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:11.934795 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:11.935836 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0926 16:47:11.936061 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4646,"thread_start_us":4435,"threads_started":1}
I20260926 16:47:11.936344 24763 tablet_service.cc:1467] Processing CreateTablet for tablet dfc202fd14134bf49a7833414e80941b (DEFAULT_TABLE table=test-table [id=c80de1adaf454da9834b43f41f05bcf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:47:11.936611 24895 tablet_service.cc:1467] Processing CreateTablet for tablet dfc202fd14134bf49a7833414e80941b (DEFAULT_TABLE table=test-table [id=c80de1adaf454da9834b43f41f05bcf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:47:11.938347 24895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfc202fd14134bf49a7833414e80941b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:47:11.938346 24763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfc202fd14134bf49a7833414e80941b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:47:11.941175 25026 tablet_service.cc:1467] Processing CreateTablet for tablet dfc202fd14134bf49a7833414e80941b (DEFAULT_TABLE table=test-table [id=c80de1adaf454da9834b43f41f05bcf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:47:11.942808 25026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfc202fd14134bf49a7833414e80941b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:47:11.954526 25108 tablet_bootstrap.cc:492] T dfc202fd14134bf49a7833414e80941b P 0685cb7ee46548989fb7a170b111181a: Bootstrap starting.
I20260926 16:47:11.957645 25109 tablet_bootstrap.cc:492] T dfc202fd14134bf49a7833414e80941b P 9a83f18a7e554e8ba00204ce1550a273: Bootstrap starting.
I20260926 16:47:11.959892 25108 tablet_bootstrap.cc:654] T dfc202fd14134bf49a7833414e80941b P 0685cb7ee46548989fb7a170b111181a: Neither blocks nor log segments found. Creating new log.
I20260926 16:47:11.960666 25110 tablet_bootstrap.cc:492] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da: Bootstrap starting.
I20260926 16:47:11.961665 25109 tablet_bootstrap.cc:654] T dfc202fd14134bf49a7833414e80941b P 9a83f18a7e554e8ba00204ce1550a273: Neither blocks nor log segments found. Creating new log.
I20260926 16:47:11.962872 25108 log.cc:824] T dfc202fd14134bf49a7833414e80941b P 0685cb7ee46548989fb7a170b111181a: Log is configured to *not* fsync() on all Append() calls
I20260926 16:47:11.962878 25109 log.cc:824] T dfc202fd14134bf49a7833414e80941b P 9a83f18a7e554e8ba00204ce1550a273: Log is configured to *not* fsync() on all Append() calls
I20260926 16:47:11.965327 25110 tablet_bootstrap.cc:654] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da: Neither blocks nor log segments found. Creating new log.
I20260926 16:47:11.966681 25108 tablet_bootstrap.cc:492] T dfc202fd14134bf49a7833414e80941b P 0685cb7ee46548989fb7a170b111181a: No bootstrap required, opened a new log
I20260926 16:47:11.966750 25110 log.cc:824] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da: Log is configured to *not* fsync() on all Append() calls
I20260926 16:47:11.967048 25108 ts_tablet_manager.cc:1397] T dfc202fd14134bf49a7833414e80941b P 0685cb7ee46548989fb7a170b111181a: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.003s
I20260926 16:47:11.967300 25109 tablet_bootstrap.cc:492] T dfc202fd14134bf49a7833414e80941b P 9a83f18a7e554e8ba00204ce1550a273: No bootstrap required, opened a new log
I20260926 16:47:11.967734 25109 ts_tablet_manager.cc:1397] T dfc202fd14134bf49a7833414e80941b P 9a83f18a7e554e8ba00204ce1550a273: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s
I20260926 16:47:11.970834 25110 tablet_bootstrap.cc:492] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da: No bootstrap required, opened a new log
I20260926 16:47:11.971201 25110 ts_tablet_manager.cc:1397] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260926 16:47:11.977229 25108 raft_consensus.cc:348] T dfc202fd14134bf49a7833414e80941b P 0685cb7ee46548989fb7a170b111181a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a83f18a7e554e8ba00204ce1550a273" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 45243 } } peers { permanent_uuid: "0685cb7ee46548989fb7a170b111181a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 35439 } } peers { permanent_uuid: "fde78bbc9fed462a855a04094fde18da" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 38291 } }
I20260926 16:47:11.977783 25108 raft_consensus.cc:374] T dfc202fd14134bf49a7833414e80941b P 0685cb7ee46548989fb7a170b111181a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:47:11.977896 25108 raft_consensus.cc:729] T dfc202fd14134bf49a7833414e80941b P 0685cb7ee46548989fb7a170b111181a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0685cb7ee46548989fb7a170b111181a, State: Initialized, Role: FOLLOWER
I20260926 16:47:11.978660 25108 consensus_queue.cc:260] T dfc202fd14134bf49a7833414e80941b P 0685cb7ee46548989fb7a170b111181a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a83f18a7e554e8ba00204ce1550a273" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 45243 } } peers { permanent_uuid: "0685cb7ee46548989fb7a170b111181a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 35439 } } peers { permanent_uuid: "fde78bbc9fed462a855a04094fde18da" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 38291 } }
I20260926 16:47:11.979734 25109 raft_consensus.cc:348] T dfc202fd14134bf49a7833414e80941b P 9a83f18a7e554e8ba00204ce1550a273 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a83f18a7e554e8ba00204ce1550a273" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 45243 } } peers { permanent_uuid: "0685cb7ee46548989fb7a170b111181a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 35439 } } peers { permanent_uuid: "fde78bbc9fed462a855a04094fde18da" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 38291 } }
I20260926 16:47:11.980149 24960 heartbeater.cc:503] Master 127.9.238.254:46415 was elected leader, sending a full tablet report...
I20260926 16:47:11.980294 25109 raft_consensus.cc:374] T dfc202fd14134bf49a7833414e80941b P 9a83f18a7e554e8ba00204ce1550a273 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:47:11.980391 25109 raft_consensus.cc:729] T dfc202fd14134bf49a7833414e80941b P 9a83f18a7e554e8ba00204ce1550a273 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a83f18a7e554e8ba00204ce1550a273, State: Initialized, Role: FOLLOWER
I20260926 16:47:11.980973 25108 ts_tablet_manager.cc:1428] T dfc202fd14134bf49a7833414e80941b P 0685cb7ee46548989fb7a170b111181a: Time spent starting tablet: real 0.014s user 0.008s sys 0.007s
I20260926 16:47:11.981114 25109 consensus_queue.cc:260] T dfc202fd14134bf49a7833414e80941b P 9a83f18a7e554e8ba00204ce1550a273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a83f18a7e554e8ba00204ce1550a273" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 45243 } } peers { permanent_uuid: "0685cb7ee46548989fb7a170b111181a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 35439 } } peers { permanent_uuid: "fde78bbc9fed462a855a04094fde18da" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 38291 } }
I20260926 16:47:11.983584 25110 raft_consensus.cc:348] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a83f18a7e554e8ba00204ce1550a273" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 45243 } } peers { permanent_uuid: "0685cb7ee46548989fb7a170b111181a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 35439 } } peers { permanent_uuid: "fde78bbc9fed462a855a04094fde18da" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 38291 } }
I20260926 16:47:11.984082 25110 raft_consensus.cc:374] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:47:11.984171 25110 raft_consensus.cc:729] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde78bbc9fed462a855a04094fde18da, State: Initialized, Role: FOLLOWER
I20260926 16:47:11.984990 25109 ts_tablet_manager.cc:1428] T dfc202fd14134bf49a7833414e80941b P 9a83f18a7e554e8ba00204ce1550a273: Time spent starting tablet: real 0.017s user 0.015s sys 0.001s
I20260926 16:47:11.984839 25110 consensus_queue.cc:260] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a83f18a7e554e8ba00204ce1550a273" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 45243 } } peers { permanent_uuid: "0685cb7ee46548989fb7a170b111181a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 35439 } } peers { permanent_uuid: "fde78bbc9fed462a855a04094fde18da" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 38291 } }
I20260926 16:47:11.986136 25091 heartbeater.cc:503] Master 127.9.238.254:46415 was elected leader, sending a full tablet report...
I20260926 16:47:11.986726 25110 ts_tablet_manager.cc:1428] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
W20260926 16:47:12.033634 24829 tablet.cc:2369] T dfc202fd14134bf49a7833414e80941b P 9a83f18a7e554e8ba00204ce1550a273: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:47:12.039961 24961 tablet.cc:2369] T dfc202fd14134bf49a7833414e80941b P 0685cb7ee46548989fb7a170b111181a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:47:12.071291 25092 tablet.cc:2369] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:47:12.077106 25116 raft_consensus.cc:482] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:47:12.077500 25116 raft_consensus.cc:504] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a83f18a7e554e8ba00204ce1550a273" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 45243 } } peers { permanent_uuid: "0685cb7ee46548989fb7a170b111181a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 35439 } } peers { permanent_uuid: "fde78bbc9fed462a855a04094fde18da" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 38291 } }
I20260926 16:47:12.079455 25116 leader_election.cc:288] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a83f18a7e554e8ba00204ce1550a273 (127.9.238.193:45243), 0685cb7ee46548989fb7a170b111181a (127.9.238.194:35439)
I20260926 16:47:12.087318 25094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:12.080002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:45243 (local address 127.9.238.195:35759)
0926 16:47:12.080295 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:12.080318 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:12.080482 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:12.080996 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:12.081002 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:12.081038 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:47:12.081279 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:12.081290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:12.082830 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:12.082838 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:12.084601 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:12.084625 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:12.086401 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:12.086408 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:12.086641 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:12.086679 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 16:47:12.086807 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 16:47:12.086993 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":1}
I20260926 16:47:12.087317 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:12.080248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:35759 (local address 127.9.238.193:45243)
0926 16:47:12.080512 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:12.080517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:12.080540 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:47:12.080663 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:12.080672 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:12.080789 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:12.080934 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:47:12.081464 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:12.082658 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:12.084837 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:12.086197 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:12.086418 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:12.086522 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:12.086839 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 16:47:12.086992 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 16:47:12.088799 25117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:12.080029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:35439 (local address 127.9.238.195:46025)
0926 16:47:12.081282 (+ 1253us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:12.081300 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:12.081481 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:12.081912 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:12.081917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:12.081946 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:47:12.082321 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:12.082333 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:12.083824 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:12.083828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:12.085913 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:12.085930 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:12.087811 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:12.087818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:12.087960 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:12.087983 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 16:47:12.088103 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 16:47:12.088243 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1116,"mutex_wait_us":283,"thread_start_us":467,"threads_started":1}
I20260926 16:47:12.089098 25105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:12.080038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:46025 (local address 127.9.238.194:35439)
0926 16:47:12.080290 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:12.080297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:12.081423 (+ 1126us) server_negotiation.cc:407] Connection header received
0926 16:47:12.081678 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:12.081682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:12.081753 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:12.081838 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:47:12.082515 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:12.083645 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:12.086165 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:12.087355 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:12.087625 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:12.087703 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:12.088159 (+ 456us) server_negotiation.cc:299] Negotiation successful
0926 16:47:12.088759 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 16:47:12.088733 24783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc202fd14134bf49a7833414e80941b" candidate_uuid: "fde78bbc9fed462a855a04094fde18da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a83f18a7e554e8ba00204ce1550a273" is_pre_election: true
I20260926 16:47:12.090451 24783 raft_consensus.cc:2454] T dfc202fd14134bf49a7833414e80941b P 9a83f18a7e554e8ba00204ce1550a273 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde78bbc9fed462a855a04094fde18da in term 0.
I20260926 16:47:12.090461 24915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc202fd14134bf49a7833414e80941b" candidate_uuid: "fde78bbc9fed462a855a04094fde18da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0685cb7ee46548989fb7a170b111181a" is_pre_election: true
I20260926 16:47:12.091162 24915 raft_consensus.cc:2454] T dfc202fd14134bf49a7833414e80941b P 0685cb7ee46548989fb7a170b111181a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde78bbc9fed462a855a04094fde18da in term 0.
I20260926 16:47:12.091912 24982 leader_election.cc:302] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a83f18a7e554e8ba00204ce1550a273, fde78bbc9fed462a855a04094fde18da; no voters:
I20260926 16:47:12.093029 25116 raft_consensus.cc:2781] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:47:12.093185 25116 raft_consensus.cc:482] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:47:12.093266 25116 raft_consensus.cc:3037] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:47:12.095456 25116 raft_consensus.cc:504] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a83f18a7e554e8ba00204ce1550a273" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 45243 } } peers { permanent_uuid: "0685cb7ee46548989fb7a170b111181a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 35439 } } peers { permanent_uuid: "fde78bbc9fed462a855a04094fde18da" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 38291 } }
I20260926 16:47:12.096217 25116 leader_election.cc:288] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da [CANDIDATE]: Term 1 election: Requested vote from peers 9a83f18a7e554e8ba00204ce1550a273 (127.9.238.193:45243), 0685cb7ee46548989fb7a170b111181a (127.9.238.194:35439)
I20260926 16:47:12.097250 24915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc202fd14134bf49a7833414e80941b" candidate_uuid: "fde78bbc9fed462a855a04094fde18da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0685cb7ee46548989fb7a170b111181a"
I20260926 16:47:12.097251 24783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc202fd14134bf49a7833414e80941b" candidate_uuid: "fde78bbc9fed462a855a04094fde18da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a83f18a7e554e8ba00204ce1550a273"
I20260926 16:47:12.097589 24783 raft_consensus.cc:3037] T dfc202fd14134bf49a7833414e80941b P 9a83f18a7e554e8ba00204ce1550a273 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:47:12.097594 24915 raft_consensus.cc:3037] T dfc202fd14134bf49a7833414e80941b P 0685cb7ee46548989fb7a170b111181a [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:47:12.101536 24915 raft_consensus.cc:2454] T dfc202fd14134bf49a7833414e80941b P 0685cb7ee46548989fb7a170b111181a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde78bbc9fed462a855a04094fde18da in term 1.
I20260926 16:47:12.101534 24783 raft_consensus.cc:2454] T dfc202fd14134bf49a7833414e80941b P 9a83f18a7e554e8ba00204ce1550a273 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde78bbc9fed462a855a04094fde18da in term 1.
I20260926 16:47:12.102459 24981 leader_election.cc:302] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0685cb7ee46548989fb7a170b111181a, 9a83f18a7e554e8ba00204ce1550a273, fde78bbc9fed462a855a04094fde18da; no voters:
I20260926 16:47:12.103094 25116 raft_consensus.cc:2781] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:47:12.103956 25116 raft_consensus.cc:686] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da [term 1 LEADER]: Becoming Leader. State: Replica: fde78bbc9fed462a855a04094fde18da, State: Running, Role: LEADER
I20260926 16:47:12.104570 25116 consensus_queue.cc:237] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a83f18a7e554e8ba00204ce1550a273" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 45243 } } peers { permanent_uuid: "0685cb7ee46548989fb7a170b111181a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 35439 } } peers { permanent_uuid: "fde78bbc9fed462a855a04094fde18da" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 38291 } }
I20260926 16:47:12.110687 24642 catalog_manager.cc:5495] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da reported cstate change: term changed from 0 to 1, leader changed from <none> to fde78bbc9fed462a855a04094fde18da (127.9.238.195). New cstate: current_term: 1 leader_uuid: "fde78bbc9fed462a855a04094fde18da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a83f18a7e554e8ba00204ce1550a273" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 45243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0685cb7ee46548989fb7a170b111181a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 35439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fde78bbc9fed462a855a04094fde18da" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 38291 } health_report { overall_health: HEALTHY } } }
I20260926 16:47:12.145093 25106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:12.138913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35500 (local address 127.9.238.195:38291)
0926 16:47:12.139184 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:12.139189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:12.139211 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:47:12.139319 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:12.139323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:12.139420 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:12.139683 (+ 263us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:47:12.140145 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:12.140875 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:12.142503 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:12.142955 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:12.143131 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:12.143701 (+ 570us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:47:12.144366 (+ 665us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:47:12.144499 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:12.144715 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 16:47:12.144808 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260926 16:47:12.156399 24783 raft_consensus.cc:1261] T dfc202fd14134bf49a7833414e80941b P 9a83f18a7e554e8ba00204ce1550a273 [term 1 FOLLOWER]: Refusing update from remote peer fde78bbc9fed462a855a04094fde18da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:47:12.156456 24915 raft_consensus.cc:1261] T dfc202fd14134bf49a7833414e80941b P 0685cb7ee46548989fb7a170b111181a [term 1 FOLLOWER]: Refusing update from remote peer fde78bbc9fed462a855a04094fde18da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:47:12.157924 25118 consensus_queue.cc:1035] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0685cb7ee46548989fb7a170b111181a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 35439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:47:12.158509 25116 consensus_queue.cc:1035] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a83f18a7e554e8ba00204ce1550a273" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 45243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:47:12.190115 24643 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:47572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:47:12.190536 24643 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:47572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:47:12.195235 24643 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad: Sending DeleteTablet for 3 replicas of tablet dfc202fd14134bf49a7833414e80941b
I20260926 16:47:12.197021 24895 tablet_service.cc:1514] Processing DeleteTablet for tablet dfc202fd14134bf49a7833414e80941b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:47:12 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:51990
I20260926 16:47:12.197108 24763 tablet_service.cc:1514] Processing DeleteTablet for tablet dfc202fd14134bf49a7833414e80941b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:47:12 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:35834
I20260926 16:47:12.197119 25026 tablet_service.cc:1514] Processing DeleteTablet for tablet dfc202fd14134bf49a7833414e80941b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:47:12 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:35486
I20260926 16:47:12.198663 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 24680
I20260926 16:47:12.208335 25131 tablet_replica.cc:330] T dfc202fd14134bf49a7833414e80941b P 9a83f18a7e554e8ba00204ce1550a273: stopping tablet replica
I20260926 16:47:12.210170 25131 raft_consensus.cc:2229] T dfc202fd14134bf49a7833414e80941b P 9a83f18a7e554e8ba00204ce1550a273 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:47:12.211930 25131 raft_consensus.cc:2258] T dfc202fd14134bf49a7833414e80941b P 9a83f18a7e554e8ba00204ce1550a273 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:47:12.216398 25131 ts_tablet_manager.cc:1905] T dfc202fd14134bf49a7833414e80941b P 9a83f18a7e554e8ba00204ce1550a273: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:47:12.219000 25132 tablet_replica.cc:330] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da: stopping tablet replica
I20260926 16:47:12.226768 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:12.199789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35836 (local address 127.9.238.193:45243)
0926 16:47:12.200770 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:12.200779 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:12.200806 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:47:12.200899 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:12.200909 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:12.201028 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:12.201161 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:12.204082 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:12.205775 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:12.208626 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:12.209582 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:12.209736 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:12.215430 (+ 5694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:12.215448 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:12.215473 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:12.215587 (+ 114us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:12.220598 (+ 5011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:12.221979 (+ 1381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:12.221992 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:12.221998 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:12.222235 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:12.222825 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:12.222831 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:12.222835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:12.223456 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:12.223878 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:12.225756 (+ 1878us) server_negotiation.cc:299] Negotiation successful
0926 16:47:12.226135 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":794}
I20260926 16:47:12.228749 24823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:47:12.229223 25130 tablet_replica.cc:330] T dfc202fd14134bf49a7833414e80941b P 0685cb7ee46548989fb7a170b111181a: stopping tablet replica
I20260926 16:47:12.229832 25132 raft_consensus.cc:2229] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:47:12.229899 25130 raft_consensus.cc:2229] T dfc202fd14134bf49a7833414e80941b P 0685cb7ee46548989fb7a170b111181a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:47:12.230829 25130 raft_consensus.cc:2258] T dfc202fd14134bf49a7833414e80941b P 0685cb7ee46548989fb7a170b111181a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:47:12.230990 25132 raft_consensus.cc:2258] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:47:12.234102 25130 ts_tablet_manager.cc:1905] T dfc202fd14134bf49a7833414e80941b P 0685cb7ee46548989fb7a170b111181a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:47:12.234216 25132 ts_tablet_manager.cc:1905] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:47:12.246754 25132 ts_tablet_manager.cc:1918] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:47:12.247228 25132 log.cc:1196] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/wal/wals/dfc202fd14134bf49a7833414e80941b
I20260926 16:47:12.248474 25132 ts_tablet_manager.cc:1939] T dfc202fd14134bf49a7833414e80941b P fde78bbc9fed462a855a04094fde18da: Deleting consensus metadata
I20260926 16:47:12.249074 25130 ts_tablet_manager.cc:1918] T dfc202fd14134bf49a7833414e80941b P 0685cb7ee46548989fb7a170b111181a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:47:12.249382 25130 log.cc:1196] T dfc202fd14134bf49a7833414e80941b P 0685cb7ee46548989fb7a170b111181a: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/wal/wals/dfc202fd14134bf49a7833414e80941b
I20260926 16:47:12.250103 25130 ts_tablet_manager.cc:1939] T dfc202fd14134bf49a7833414e80941b P 0685cb7ee46548989fb7a170b111181a: Deleting consensus metadata
I20260926 16:47:12.251253 24630 catalog_manager.cc:4841] TS fde78bbc9fed462a855a04094fde18da (127.9.238.195:38291): tablet dfc202fd14134bf49a7833414e80941b (table test-table [id=c80de1adaf454da9834b43f41f05bcf4]) successfully deleted
I20260926 16:47:12.252409 24629 catalog_manager.cc:4841] TS 0685cb7ee46548989fb7a170b111181a (127.9.238.194:35439): tablet dfc202fd14134bf49a7833414e80941b (table test-table [id=c80de1adaf454da9834b43f41f05bcf4]) successfully deleted
I20260926 16:47:12.372813 25131 ts_tablet_manager.cc:1918] T dfc202fd14134bf49a7833414e80941b P 9a83f18a7e554e8ba00204ce1550a273: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:47:12.373199 25131 log.cc:1196] T dfc202fd14134bf49a7833414e80941b P 9a83f18a7e554e8ba00204ce1550a273: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/wal/wals/dfc202fd14134bf49a7833414e80941b
I20260926 16:47:12.373687 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 24680
I20260926 16:47:12.397405 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 24832
W20260926 16:47:12.397353 24630 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.238.193:45243 (error 108)
I20260926 16:47:12.399878 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:12.399223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:45243 (local address 127.0.0.1:35846)
0926 16:47:12.399519 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:12.399692 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:45243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260926 16:47:12.400219 24630 catalog_manager.cc:4584] TS 9a83f18a7e554e8ba00204ce1550a273 (127.9.238.193:45243): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dfc202fd14134bf49a7833414e80941b: Network error: Client connection negotiation failed: client connection to 127.9.238.193:45243: connect: Connection refused (error 111)
I20260926 16:47:12.417532 25105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:12.398665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52004 (local address 127.9.238.194:35439)
0926 16:47:12.398878 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:12.398884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:12.398910 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:47:12.399038 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:12.399044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:12.399141 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:12.399277 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:12.401565 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:12.402494 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:12.404818 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:12.405339 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:12.405497 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:12.409512 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:12.409535 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:12.409557 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:12.409629 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:12.413598 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:12.414681 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:12.414692 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:12.414695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:12.414880 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:12.415444 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:12.415448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:12.415450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:12.416150 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:12.416429 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:12.416843 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 16:47:12.417068 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 16:47:12.418543 24955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:47:12.454438 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:12.453716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:45243 (local address 127.0.0.1:35858)
0926 16:47:12.454066 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:12.454237 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:45243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260926 16:47:12.456111 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:12.455471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:45243 (local address 127.0.0.1:35874)
0926 16:47:12.455722 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:12.455884 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:45243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 16:47:12.517396 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:12.516532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:45243 (local address 127.0.0.1:35878)
0926 16:47:12.516940 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:12.517117 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:45243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260926 16:47:12.520505 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:12.519936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:45243 (local address 127.0.0.1:35882)
0926 16:47:12.520234 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:12.520338 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:45243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260926 16:47:12.520812 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 24832
I20260926 16:47:12.550148 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 24963
I20260926 16:47:12.570394 25106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:12.551053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35512 (local address 127.9.238.195:38291)
0926 16:47:12.551312 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:12.551316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:12.551338 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:47:12.551682 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:12.551686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:12.551774 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:12.551872 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:12.554052 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:12.554917 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:12.556982 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:12.557606 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:12.557748 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:12.561919 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:12.561938 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:12.561957 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:12.562019 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:12.566266 (+ 4247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:12.567688 (+ 1422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:12.567696 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:12.567699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:12.567903 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:12.568488 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:12.568494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:12.568496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:12.568935 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:12.569200 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:12.569731 (+ 531us) server_negotiation.cc:299] Negotiation successful
0926 16:47:12.569933 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260926 16:47:12.571435 25086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:47:12.603134 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:12.602465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:45243 (local address 127.0.0.1:35884)
0926 16:47:12.602786 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:12.602908 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:45243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260926 16:47:12.604367 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:12.603955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:45243 (local address 127.0.0.1:35896)
0926 16:47:12.604166 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:12.604249 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:45243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260926 16:47:12.706406 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 24963
I20260926 16:47:12.728263 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 24610
I20260926 16:47:12.729943 24673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:47:12.869313 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:12.868561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:45243 (local address 127.0.0.1:35902)
0926 16:47:12.868875 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:12.869024 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:45243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260926 16:47:12.869655 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 24610
I20260926 16:47:12.899422 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:46415
--webserver_interface=127.9.238.254
--webserver_port=34423
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:36335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:46415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:47:13.265324 25138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:47:13.265705 25138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:47:13.266016 25138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:47:13.274514 25138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:47:13.274669 25138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:47:13.274751 25138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:47:13.274847 25138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:47:13.274935 25138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:47:13.285571 25138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:36335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:46415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:46415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=34423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:47:13.287045 25138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:47:13.288443 25138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:47:13.302614 25143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:13.302605 25144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:13.302721 25146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:47:13.303779 25138 server_base.cc:1034] running on GCE node
I20260926 16:47:13.305018 25138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:47:13.307358 25138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:47:13.308724 25138 hybrid_clock.cc:648] HybridClock initialized: now 1790441233308620 us; error 66 us; skew 500 ppm
I20260926 16:47:13.309309 25138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:47:13 dist-test-slave-wt2j krb5kdc[24058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441233, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:47:13.313792 25138 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:47:13.316121 25138 webserver.cc:460] Webserver started at http://127.9.238.254:34423/ using document root <none> and password file <none>
I20260926 16:47:13.317157 25138 fs_manager.cc:359] Metadata directory not provided
I20260926 16:47:13.317349 25138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:47:13.324928 25138 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260926 16:47:13.329200 25153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:13.331676 25138 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.003s
I20260926 16:47:13.332005 25138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "2a81ad67dca84ffabddd19f5d60e3cad"
format_stamp: "Formatted at 2026-09-26 16:47:06 on dist-test-slave-wt2j"
server_key: "c419107dd2a6a00cced3eb5c7ff9c412"
server_key_iv: "049fca501495e8e60d3582a1b40ec1c4"
server_key_version: "encryptionkey@0"
I20260926 16:47:13.333372 25138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:47:13.367879 25138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:47:13.374450 25138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:47:13.374836 25138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:47:13.400792 25138 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:46415
I20260926 16:47:13.400808 25204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:46415 every 8 connection(s)
I20260926 16:47:13.402930 25138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:47:13.410948 25205 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:47:13.412226 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 25138
I20260926 16:47:13.413471 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:45243
--local_ip_for_outbound_sockets=127.9.238.193
--tserver_master_addrs=127.9.238.254:46415
--webserver_port=39237
--webserver_interface=127.9.238.193
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:36335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:47:13.414738 25205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad: Bootstrap starting.
I20260926 16:47:13.436836 25205 log.cc:824] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad: Log is configured to *not* fsync() on all Append() calls
I20260926 16:47:13.453977 25205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:47:13.454608 25205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad: Bootstrap complete.
I20260926 16:47:13.464591 25205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a81ad67dca84ffabddd19f5d60e3cad" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46415 } }
I20260926 16:47:13.465628 25205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a81ad67dca84ffabddd19f5d60e3cad, State: Initialized, Role: FOLLOWER
I20260926 16:47:13.466360 25205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a81ad67dca84ffabddd19f5d60e3cad" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46415 } }
I20260926 16:47:13.466645 25205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:47:13.466789 25205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:47:13.466989 25205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [term 2 FOLLOWER]: Advancing to term 3
I20260926 16:47:13.471293 25205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a81ad67dca84ffabddd19f5d60e3cad" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46415 } }
I20260926 16:47:13.471971 25205 leader_election.cc:302] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a81ad67dca84ffabddd19f5d60e3cad; no voters:
I20260926 16:47:13.472743 25205 leader_election.cc:288] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 16:47:13.472883 25209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [term 3 FOLLOWER]: Leader election won for term 3
I20260926 16:47:13.474795 25209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [term 3 LEADER]: Becoming Leader. State: Replica: 2a81ad67dca84ffabddd19f5d60e3cad, State: Running, Role: LEADER
I20260926 16:47:13.475811 25209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a81ad67dca84ffabddd19f5d60e3cad" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46415 } }
I20260926 16:47:13.477370 25205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:47:13.479215 25210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a81ad67dca84ffabddd19f5d60e3cad. Latest consensus state: current_term: 3 leader_uuid: "2a81ad67dca84ffabddd19f5d60e3cad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a81ad67dca84ffabddd19f5d60e3cad" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46415 } } }
I20260926 16:47:13.479869 25210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [sys.catalog]: This master's current role is: LEADER
I20260926 16:47:13.480804 25211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2a81ad67dca84ffabddd19f5d60e3cad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a81ad67dca84ffabddd19f5d60e3cad" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46415 } } }
I20260926 16:47:13.481197 25211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a81ad67dca84ffabddd19f5d60e3cad [sys.catalog]: This master's current role is: LEADER
I20260926 16:47:13.488986 25214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:47:13.501118 25214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:47:13.503413 25214 catalog_manager.cc:1259] Loaded cluster ID: 58ed3eb0b84b4b2b804dc006c690be83
I20260926 16:47:13.503669 25214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 16:47:13.506868 25214 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f25e1a0e9ad google::LogMessage::Fail() at ??:0
@ 0x7f25e1a1188a google::LogMessage::SendToLog() at ??:0
@ 0x7f25e1a0e4de google::LogMessage::Flush() at ??:0
@ 0x7f25e1a0fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f25f7441512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f25f743e976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f25f74c6610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f25f74c6439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f25e465a345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f25e465d91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f25e465d799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f25e4638eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f25e79126db start_thread at ??:0
@ 0x7f25e08a861f clone at ??:0
W20260926 16:47:13.865924 25207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:47:13.866271 25207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:47:13.866382 25207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:47:13.874451 25207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:47:13.874578 25207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:47:13.874701 25207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:47:13.885708 25207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:36335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:45243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=39237
--tserver_master_addrs=127.9.238.254:46415
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:47:13.887055 25207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:47:13.888515 25207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:47:13.903846 25246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:13.904404 25249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:13.904575 25247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:47:13.905273 25207 server_base.cc:1034] running on GCE node
I20260926 16:47:13.906266 25207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:47:13.907691 25207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:47:13.909032 25207 hybrid_clock.cc:648] HybridClock initialized: now 1790441233908940 us; error 66 us; skew 500 ppm
I20260926 16:47:13.909610 25207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:47:13 dist-test-slave-wt2j krb5kdc[24058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441233, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:47:13.913884 25207 init.cc:377] Logged in from keytab as kudu/127.9.238.193@KRBTEST.COM (short username kudu)
I20260926 16:47:13.915907 25207 webserver.cc:460] Webserver started at http://127.9.238.193:39237/ using document root <none> and password file <none>
I20260926 16:47:13.916843 25207 fs_manager.cc:359] Metadata directory not provided
I20260926 16:47:13.917016 25207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:47:13.924345 25207 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.002s
I20260926 16:47:13.928560 25256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:13.930780 25207 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260926 16:47:13.931093 25207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "9a83f18a7e554e8ba00204ce1550a273"
format_stamp: "Formatted at 2026-09-26 16:47:07 on dist-test-slave-wt2j"
server_key: "6d01d3f70319ed0ddcf6826a5ce62ace"
server_key_iv: "95742a9c60913135bbf7407dd93a1092"
server_key_version: "encryptionkey@0"
I20260926 16:47:13.932535 25207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:47:13.954916 25207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:47:13.961103 25207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:47:13.961485 25207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:47:13.962833 25207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:47:13.965444 25263 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260926 16:47:13.971318 25263 ts_tablet_manager.cc:1852] T dfc202fd14134bf49a7833414e80941b P 9a83f18a7e554e8ba00204ce1550a273: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 16:47:13.972431 25263 ts_tablet_manager.cc:1905] T dfc202fd14134bf49a7833414e80941b P 9a83f18a7e554e8ba00204ce1550a273: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:47:13.980391 25263 ts_tablet_manager.cc:1918] T dfc202fd14134bf49a7833414e80941b P 9a83f18a7e554e8ba00204ce1550a273: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:47:13.980708 25263 ts_tablet_manager.cc:1939] T dfc202fd14134bf49a7833414e80941b P 9a83f18a7e554e8ba00204ce1550a273: Deleting consensus metadata
I20260926 16:47:13.981146 25207 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 16:47:13.981389 25207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.001s sys 0.000s
I20260926 16:47:13.981606 25207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:47:13.981760 25207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:14.032018 25207 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:45243
I20260926 16:47:14.032133 25369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:45243 every 8 connection(s)
I20260926 16:47:14.033779 25207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:47:14.037685 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 25207
I20260926 16:47:14.038573 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:35439
--local_ip_for_outbound_sockets=127.9.238.194
--tserver_master_addrs=127.9.238.254:46415
--webserver_port=40805
--webserver_interface=127.9.238.194
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:36335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:47:14.040727 25372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:14.037775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.193:59109)
0926 16:47:14.039011 (+ 1236us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:14.039404 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":477,"threads_started":1}
W20260926 16:47:14.041725 25370 heartbeater.cc:644] Failed to heartbeat to 127.9.238.254:46415 (0 consecutive failures): Network error: Failed to ping master at 127.9.238.254:46415: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
I20260926 16:47:14.042881 25372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:14.042447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.193:59121)
0926 16:47:14.042646 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:14.042738 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 16:47:14.043893 25372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:14.043486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.193:37257)
0926 16:47:14.043673 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:14.043762 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260926 16:47:14.044152 25370 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 16:47:14.390504 25373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:47:14.390815 25373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:47:14.390966 25373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:47:14.398844 25373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:47:14.398993 25373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:47:14.399188 25373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:47:14.409356 25373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:36335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:35439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=40805
--tserver_master_addrs=127.9.238.254:46415
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:47:14.410758 25373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:47:14.412312 25373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:47:14.427481 25378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:14.427618 25381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:14.427608 25379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:47:14.429474 25373 server_base.cc:1034] running on GCE node
I20260926 16:47:14.430234 25373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:47:14.431428 25373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:47:14.432649 25373 hybrid_clock.cc:648] HybridClock initialized: now 1790441234432586 us; error 60 us; skew 500 ppm
I20260926 16:47:14.433010 25373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:47:14 dist-test-slave-wt2j krb5kdc[24058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441234, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:47:14.436738 25373 init.cc:377] Logged in from keytab as kudu/127.9.238.194@KRBTEST.COM (short username kudu)
I20260926 16:47:14.438731 25373 webserver.cc:460] Webserver started at http://127.9.238.194:40805/ using document root <none> and password file <none>
I20260926 16:47:14.439503 25373 fs_manager.cc:359] Metadata directory not provided
I20260926 16:47:14.439755 25373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:47:14.446621 25373 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 16:47:14.450320 25388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:14.452319 25373 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 16:47:14.452597 25373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "0685cb7ee46548989fb7a170b111181a"
format_stamp: "Formatted at 2026-09-26 16:47:07 on dist-test-slave-wt2j"
server_key: "1691e2ee85552ffcd7a026cacd9a0501"
server_key_iv: "ecb1b09a3cb01e428f4cde3b2287f1f0"
server_key_version: "encryptionkey@0"
I20260926 16:47:14.453693 25373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:47:14.504593 25373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:47:14.510056 25373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:47:14.510432 25373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:47:14.511627 25373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:47:14.513242 25373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:47:14.513377 25373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:14.513527 25373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:47:14.513628 25373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:14.556705 25373 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:35439
I20260926 16:47:14.556725 25500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:35439 every 8 connection(s)
I20260926 16:47:14.558429 25373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:47:14.565331 25503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:14.562403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.194:54195)
0926 16:47:14.563516 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:14.563973 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":393,"threads_started":1}
W20260926 16:47:14.566480 25501 heartbeater.cc:644] Failed to heartbeat to 127.9.238.254:46415 (0 consecutive failures): Network error: Failed to ping master at 127.9.238.254:46415: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
I20260926 16:47:14.567413 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 25373
I20260926 16:47:14.567931 25503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:14.567306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.194:45535)
0926 16:47:14.567624 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:14.567744 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260926 16:47:14.568377 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:38291
--local_ip_for_outbound_sockets=127.9.238.195
--tserver_master_addrs=127.9.238.254:46415
--webserver_port=36199
--webserver_interface=127.9.238.195
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:36335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:47:14.569257 25503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:14.568748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.194:41621)
0926 16:47:14.568950 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:14.569072 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 16:47:14.569660 25501 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 16:47:14.909598 25504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:47:14.909910 25504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:47:14.910068 25504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:47:14.917631 25504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:47:14.917764 25504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:47:14.917958 25504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:47:14.928217 25504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:36335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:38291
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=36199
--tserver_master_addrs=127.9.238.254:46415
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:47:14.929634 25504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:47:14.931010 25504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:47:14.946663 25510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:14.946687 25509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:14.946769 25512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:47:14.947782 25504 server_base.cc:1034] running on GCE node
I20260926 16:47:14.948495 25504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:47:14.949628 25504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:47:14.950907 25504 hybrid_clock.cc:648] HybridClock initialized: now 1790441234950851 us; error 46 us; skew 500 ppm
I20260926 16:47:14.951346 25504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:47:14 dist-test-slave-wt2j krb5kdc[24058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441234, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:47:14.955264 25504 init.cc:377] Logged in from keytab as kudu/127.9.238.195@KRBTEST.COM (short username kudu)
I20260926 16:47:14.957312 25504 webserver.cc:460] Webserver started at http://127.9.238.195:36199/ using document root <none> and password file <none>
I20260926 16:47:14.958101 25504 fs_manager.cc:359] Metadata directory not provided
I20260926 16:47:14.958274 25504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:47:14.964350 25504 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 16:47:14.967680 25519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:14.969293 25504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 16:47:14.969518 25504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "fde78bbc9fed462a855a04094fde18da"
format_stamp: "Formatted at 2026-09-26 16:47:08 on dist-test-slave-wt2j"
server_key: "9ac1b971a5e4869a36e58713887f9265"
server_key_iv: "f3887687ab9409bb9fddff75e8066089"
server_key_version: "encryptionkey@0"
I20260926 16:47:14.970592 25504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:47:15.000700 25504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:47:15.006047 25504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:47:15.006392 25504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:47:15.007915 25504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:47:15.009548 25504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:47:15.009688 25504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:15.009838 25504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:47:15.009935 25504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:15.046101 25615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:15.044958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.193:38093)
0926 16:47:15.045640 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:15.045801 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":303,"threads_started":1}
I20260926 16:47:15.053920 25504 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:38291
I20260926 16:47:15.054044 25632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:38291 every 8 connection(s)
I20260926 16:47:15.055666 25504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:47:15.061089 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 25504
I20260926 16:47:15.062602 25635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:15.059809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.195:41321)
0926 16:47:15.060888 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:15.061265 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":375,"threads_started":1}
W20260926 16:47:15.063673 25633 heartbeater.cc:644] Failed to heartbeat to 127.9.238.254:46415 (0 consecutive failures): Network error: Failed to ping master at 127.9.238.254:46415: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
I20260926 16:47:15.064723 25635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:15.064271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.195:41603)
0926 16:47:15.064496 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:15.064587 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 16:47:15.065757 25635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:15.065372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.195:46025)
0926 16:47:15.065556 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:15.065636 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260926 16:47:15.066041 25633 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 16:47:15.571815 25637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:15.570602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.194:38119)
0926 16:47:15.571329 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:15.571464 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":336,"threads_started":1}
I20260926 16:47:16.048523 25638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:16.047485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.193:47685)
0926 16:47:16.048168 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:16.048300 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":385,"threads_started":1}
I20260926 16:47:16.067684 25639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:16.066701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.195:57297)
0926 16:47:16.067251 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:16.067354 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":287,"threads_started":1}
I20260926 16:47:16.574818 25640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:16.573753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.194:37121)
0926 16:47:16.574517 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:16.574636 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":362,"threads_started":1}
I20260926 16:47:17.050915 25641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:17.049928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.193:38431)
0926 16:47:17.050598 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:17.050726 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":316,"threads_started":1}
I20260926 16:47:17.070158 25642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:17.069060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.195:49485)
0926 16:47:17.069801 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:17.069942 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":363,"threads_started":1}
I20260926 16:47:17.577610 25643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:17.576574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.194:46101)
0926 16:47:17.577181 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:17.577373 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":304,"threads_started":1}
I20260926 16:47:18.053263 25644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:18.052221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.193:55317)
0926 16:47:18.052907 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:18.053004 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":339,"threads_started":1}
I20260926 16:47:18.072429 25645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:18.071268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.195:60731)
0926 16:47:18.072057 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:18.072188 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":435,"threads_started":1}
I20260926 16:47:18.579901 25646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:18.578948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.194:43325)
0926 16:47:18.579601 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:18.579708 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":316,"threads_started":1}
I20260926 16:47:19.055516 25647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:19.054637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.193:40881)
0926 16:47:19.055248 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:19.055360 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":333,"threads_started":1}
I20260926 16:47:19.074985 25648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:19.073848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.195:36753)
0926 16:47:19.074610 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:19.074761 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":338,"threads_started":1}
I20260926 16:47:19.582113 25649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:19.581141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.194:37207)
0926 16:47:19.581750 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:19.581890 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":297,"threads_started":1}
I20260926 16:47:20.058144 25650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:20.056938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.193:33991)
0926 16:47:20.057736 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:20.057900 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":387,"threads_started":1}
I20260926 16:47:20.077291 25651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:20.076434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.195:36661)
0926 16:47:20.076993 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:20.077106 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":297,"threads_started":1}
I20260926 16:47:20.584489 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:20.583454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.194:38003)
0926 16:47:20.584154 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:20.584300 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":394,"threads_started":1}
I20260926 16:47:21.060410 25653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:21.059338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.193:48557)
0926 16:47:21.060077 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:21.060174 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":348,"threads_started":1}
I20260926 16:47:21.079766 25654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:21.078497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.195:38561)
0926 16:47:21.079304 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:21.079428 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":518,"threads_started":1}
I20260926 16:47:21.586755 25655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:21.585786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.194:51647)
0926 16:47:21.586414 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:21.586535 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":282,"threads_started":1}
I20260926 16:47:22.062541 25656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:22.061625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.193:39619)
0926 16:47:22.062238 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:22.062349 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":321,"threads_started":1}
I20260926 16:47:22.082278 25657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:22.081250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.195:39983)
0926 16:47:22.081919 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:22.082034 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":295,"threads_started":1}
I20260926 16:47:22.589179 25658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:22.588032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.194:41753)
0926 16:47:22.588804 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:22.588934 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":456,"threads_started":1}
I20260926 16:47:23.065094 25659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:23.064064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.193:46985)
0926 16:47:23.064795 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:23.064923 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":327,"threads_started":1}
I20260926 16:47:23.084707 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:23.083721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.195:34711)
0926 16:47:23.084411 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:23.084536 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":358,"threads_started":1}
I20260926 16:47:23.591741 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:23.590550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.194:34175)
0926 16:47:23.591262 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:23.591396 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":334,"threads_started":1}
I20260926 16:47:24.067327 25662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:24.066373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.193:41499)
0926 16:47:24.066947 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:24.067051 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":283,"threads_started":1}
I20260926 16:47:24.087026 25663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:24.086077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.195:57985)
0926 16:47:24.086704 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:24.086837 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":356,"threads_started":1}
I20260926 16:47:24.594182 25664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:24.593139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.194:38925)
0926 16:47:24.593781 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:24.593948 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":330,"threads_started":1}
I20260926 16:47:25.069576 25665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:25.068637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.193:53241)
0926 16:47:25.069248 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:25.069357 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":327,"threads_started":1}
I20260926 16:47:25.089571 25666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:25.088438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.195:47687)
0926 16:47:25.089163 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:25.089282 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":391,"threads_started":1}
I20260926 16:47:25.596837 25667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:25.595534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.194:33089)
0926 16:47:25.596325 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:25.596485 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":269,"threads_started":1}
I20260926 16:47:26.071766 25668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:26.070808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.193:49915)
0926 16:47:26.071396 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:26.071489 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":337,"threads_started":1}
I20260926 16:47:26.092069 25669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:26.090949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.195:55907)
0926 16:47:26.091700 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:26.091830 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":396,"threads_started":1}
I20260926 16:47:26.599474 25670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:26.598506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.194:43459)
0926 16:47:26.599152 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:26.599286 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":351,"threads_started":1}
I20260926 16:47:27.074208 25671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:27.072949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.193:57959)
0926 16:47:27.073750 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:27.073970 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":312,"threads_started":1}
I20260926 16:47:27.094682 25672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:27.093509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.195:56059)
0926 16:47:27.094273 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:27.094427 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":336,"threads_started":1}
I20260926 16:47:27.601946 25673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:27.600887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.194:54709)
0926 16:47:27.601571 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:27.601704 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":341,"threads_started":1}
I20260926 16:47:28.076354 25674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:28.075397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.193:43435)
0926 16:47:28.076029 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:28.076134 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":329,"threads_started":1}
I20260926 16:47:28.097021 25675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:28.095951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.195:39191)
0926 16:47:28.096656 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:28.096772 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":391,"threads_started":1}
I20260926 16:47:28.604230 25676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:28.603155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.194:57773)
0926 16:47:28.603859 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:28.603986 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":307,"threads_started":1}
I20260926 16:47:29.079042 25677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:29.077958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.193:36227)
0926 16:47:29.078703 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:29.078840 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":430,"threads_started":1}
I20260926 16:47:29.099371 25678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:29.098334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.195:36461)
0926 16:47:29.099025 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:29.099132 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":388,"threads_started":1}
I20260926 16:47:29.607206 25679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:29.606169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.194:55753)
0926 16:47:29.606807 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:29.606963 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":311,"threads_started":1}
I20260926 16:47:30.064877 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 25207
I20260926 16:47:30.081617 25681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:30.080371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.193:57065)
0926 16:47:30.081097 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:30.081323 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":349,"threads_started":1}
I20260926 16:47:30.090658 25680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:30.066379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47846 (local address 127.9.238.193:45243)
0926 16:47:30.067298 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:30.067330 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:30.067408 (+ 78us) server_negotiation.cc:407] Connection header received
0926 16:47:30.070355 (+ 2947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:30.070371 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:30.071031 (+ 660us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:30.071491 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:30.073162 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:30.074961 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:30.076618 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:30.077540 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:30.077731 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:30.081295 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:30.081305 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:30.081344 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:30.081402 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:30.085520 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:30.086496 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:30.086505 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:30.086516 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:30.086794 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:30.087356 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:30.087361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:30.087364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:30.088396 (+ 1032us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:30.088731 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:30.089695 (+ 964us) server_negotiation.cc:299] Negotiation successful
0926 16:47:30.090264 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":452,"threads_started":1}
I20260926 16:47:30.092751 25365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:47:30.102412 25683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:30.101157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46415 (local address 127.9.238.195:42653)
0926 16:47:30.101968 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:30.102126 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:46415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":402,"threads_started":1}
I20260926 16:47:30.191411 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 25207
I20260926 16:47:30.210711 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 25373
I20260926 16:47:30.241531 25684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:30.212201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39790 (local address 127.9.238.194:35439)
0926 16:47:30.212968 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:30.212987 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:30.213038 (+ 51us) server_negotiation.cc:407] Connection header received
0926 16:47:30.214979 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:30.214995 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:30.215703 (+ 708us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:30.216108 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:30.217824 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:30.219335 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:30.220984 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:30.222015 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:30.222277 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:30.232557 (+ 10280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:30.232572 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:30.232599 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:30.232672 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:30.236793 (+ 4121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:30.237885 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:30.237895 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:30.237905 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:30.238183 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:30.238678 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:30.238681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:30.238683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:30.239487 (+ 804us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:30.239927 (+ 440us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:30.240416 (+ 489us) server_negotiation.cc:299] Negotiation successful
0926 16:47:30.240997 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":365,"threads_started":1}
I20260926 16:47:30.243541 25496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:47:30.343765 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 25373
I20260926 16:47:30.366750 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 25504
I20260926 16:47:30.390004 25686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:30.368127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40406 (local address 127.9.238.195:38291)
0926 16:47:30.368967 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:30.368987 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:30.369041 (+ 54us) server_negotiation.cc:407] Connection header received
0926 16:47:30.370999 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:30.371013 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:30.371719 (+ 706us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:30.372237 (+ 518us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:30.374075 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:30.375687 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:30.377097 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:30.377898 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:30.378125 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:30.381807 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:30.381818 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:30.381849 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:30.381911 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:30.385714 (+ 3803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:30.386703 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:30.386712 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:30.386721 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:30.386937 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:30.387375 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:30.387380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:30.387384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:30.388185 (+ 801us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:30.388586 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:30.389069 (+ 483us) server_negotiation.cc:299] Negotiation successful
0926 16:47:30.389642 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":367,"threads_started":1}
I20260926 16:47:30.391846 25628 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:47:30.488791 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 25504
2026-09-26T16:47:30Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25904 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:47:30 dist-test-slave-wt2j krb5kdc[25690](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:47:30 dist-test-slave-wt2j krb5kdc[25690](info): set up 2 sockets
Sep 26 16:47:30 dist-test-slave-wt2j krb5kdc[25690](info): commencing operation
krb5kdc: starting...
W20260926 16:47:31.646173 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.003s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:47:31 dist-test-slave-wt2j krb5kdc[25690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:47:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:47:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:47:31.949056 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:35375
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:45093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:35375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:47:32.305708 25703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:47:32.306043 25703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:47:32.306226 25703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:47:32.313777 25703 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 16:47:32.313898 25703 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 16:47:32.313938 25703 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 16:47:32.314037 25703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:47:32.314134 25703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:47:32.314229 25703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:47:32.314304 25703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:47:32.314399 25703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:47:32.324757 25703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:45093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:35375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:35375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:47:32.326156 25703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:47:32.327646 25703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:47:32.341269 25709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:32.341425 25711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:32.342099 25708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:47:32.342630 25703 server_base.cc:1034] running on GCE node
I20260926 16:47:32.343540 25703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:47:32.345124 25703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:47:32.346436 25703 hybrid_clock.cc:648] HybridClock initialized: now 1790441252346358 us; error 53 us; skew 500 ppm
I20260926 16:47:32.346832 25703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:47:32 dist-test-slave-wt2j krb5kdc[25690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441252, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:47:32.350687 25703 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:47:32.352617 25703 webserver.cc:460] Webserver started at http://127.9.238.254:39433/ using document root <none> and password file <none>
I20260926 16:47:32.353420 25703 fs_manager.cc:359] Metadata directory not provided
I20260926 16:47:32.353555 25703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:47:32.353907 25703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:47:32.357304 25703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "367b3ecfd4f3444999c5e41fa619c7c4"
format_stamp: "Formatted at 2026-09-26 16:47:32 on dist-test-slave-wt2j"
server_key: "9e3f04b9ec02587a3063bbf571fe0483"
server_key_iv: "14edfba153f0ac42caa1f6136a40b942"
server_key_version: "encryptionkey@0"
I20260926 16:47:32.358153 25703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "367b3ecfd4f3444999c5e41fa619c7c4"
format_stamp: "Formatted at 2026-09-26 16:47:32 on dist-test-slave-wt2j"
server_key: "9e3f04b9ec02587a3063bbf571fe0483"
server_key_iv: "14edfba153f0ac42caa1f6136a40b942"
server_key_version: "encryptionkey@0"
I20260926 16:47:32.364161 25703 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 16:47:32.368515 25718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:32.370371 25703 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:47:32.370615 25703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "367b3ecfd4f3444999c5e41fa619c7c4"
format_stamp: "Formatted at 2026-09-26 16:47:32 on dist-test-slave-wt2j"
server_key: "9e3f04b9ec02587a3063bbf571fe0483"
server_key_iv: "14edfba153f0ac42caa1f6136a40b942"
server_key_version: "encryptionkey@0"
I20260926 16:47:32.370855 25703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:47:32.387971 25703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:47:32.392825 25703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:47:32.393225 25703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:47:32.416469 25703 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:35375
I20260926 16:47:32.416504 25769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:35375 every 8 connection(s)
I20260926 16:47:32.418195 25703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:47:32.423645 25770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:47:32.428750 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 25703
I20260926 16:47:32.429029 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:47:32.430012 10171 external_mini_cluster.cc:1428] Setting key b4152e93c62872501a4991df5bd42ea9
I20260926 16:47:32.441670 25770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4: Bootstrap starting.
I20260926 16:47:32.446873 25770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4: Neither blocks nor log segments found. Creating new log.
I20260926 16:47:32.448403 25770 log.cc:824] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4: Log is configured to *not* fsync() on all Append() calls
Sep 26 16:47:32 dist-test-slave-wt2j krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:47:32.454008 25770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4: No bootstrap required, opened a new log
I20260926 16:47:32.459373 25773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:32.432860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57574 (local address 127.9.238.254:35375)
0926 16:47:32.433840 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:32.433855 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:32.433910 (+ 55us) server_negotiation.cc:407] Connection header received
0926 16:47:32.436173 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:32.436200 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:32.436722 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:32.437146 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:32.439479 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:32.441157 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:32.443292 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:32.444232 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:32.444486 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:32.450292 (+ 5806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:32.450312 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:32.450349 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:32.450438 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:32.454595 (+ 4157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:32.455438 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:32.455447 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:32.455454 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:32.455713 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:32.456105 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:32.456116 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:32.456120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:32.456796 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:32.457174 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:32.457662 (+ 488us) server_negotiation.cc:299] Negotiation successful
0926 16:47:32.458165 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":462,"threads_started":1}
I20260926 16:47:32.463896 25770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367b3ecfd4f3444999c5e41fa619c7c4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35375 } }
I20260926 16:47:32.464430 25770 raft_consensus.cc:374] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:47:32.464540 25770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 367b3ecfd4f3444999c5e41fa619c7c4, State: Initialized, Role: FOLLOWER
I20260926 16:47:32.465241 25770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367b3ecfd4f3444999c5e41fa619c7c4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35375 } }
I20260926 16:47:32.465523 25770 raft_consensus.cc:388] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:47:32.465673 25770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:47:32.465860 25770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:47:32.468946 25770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367b3ecfd4f3444999c5e41fa619c7c4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35375 } }
I20260926 16:47:32.469591 25770 leader_election.cc:302] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 367b3ecfd4f3444999c5e41fa619c7c4; no voters:
I20260926 16:47:32.470405 25770 leader_election.cc:288] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:47:32.470566 25775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:47:32.472012 25775 raft_consensus.cc:686] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [term 1 LEADER]: Becoming Leader. State: Replica: 367b3ecfd4f3444999c5e41fa619c7c4, State: Running, Role: LEADER
I20260926 16:47:32.472867 25775 consensus_queue.cc:237] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367b3ecfd4f3444999c5e41fa619c7c4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35375 } }
I20260926 16:47:32.474155 25770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:47:32.477515 25777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 367b3ecfd4f3444999c5e41fa619c7c4. Latest consensus state: current_term: 1 leader_uuid: "367b3ecfd4f3444999c5e41fa619c7c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367b3ecfd4f3444999c5e41fa619c7c4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35375 } } }
I20260926 16:47:32.477476 25776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "367b3ecfd4f3444999c5e41fa619c7c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367b3ecfd4f3444999c5e41fa619c7c4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35375 } } }
I20260926 16:47:32.478094 25777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:47:32.478145 25776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:47:32.479854 25781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:47:32.485782 25781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:47:32.497355 25781 catalog_manager.cc:1347] Generated new cluster ID: 6fc673e8d4434f8997712dcaed43249a
I20260926 16:47:32.497512 25781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:47:32.512467 25781 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:47:32.513931 25781 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:47:32.525872 25781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4: Generated new TSK 0
I20260926 16:47:32.527390 25781 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
I20260926 16:47:32.596179 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:0
--local_ip_for_outbound_sockets=127.9.238.193
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:35375
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:45093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:47:32.962743 25796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:47:32.963056 25796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:47:32.963155 25796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:47:32.970685 25796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:47:32.970804 25796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:47:32.970921 25796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:47:32.981689 25796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:45093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:35375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:47:32.983124 25796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:47:32.984462 25796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:47:33.000874 25802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:33.000928 25804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:33.000840 25801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:47:33.001968 25796 server_base.cc:1034] running on GCE node
I20260926 16:47:33.002681 25796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:47:33.003827 25796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:47:33.005165 25796 hybrid_clock.cc:648] HybridClock initialized: now 1790441253005066 us; error 97 us; skew 500 ppm
I20260926 16:47:33.005568 25796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:47:33 dist-test-slave-wt2j krb5kdc[25690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441253, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:47:33.009616 25796 init.cc:377] Logged in from keytab as kudu/127.9.238.193@KRBTEST.COM (short username kudu)
I20260926 16:47:33.011598 25796 webserver.cc:460] Webserver started at http://127.9.238.193:37245/ using document root <none> and password file <none>
I20260926 16:47:33.012353 25796 fs_manager.cc:359] Metadata directory not provided
I20260926 16:47:33.012499 25796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:47:33.012817 25796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:47:33.015997 25796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/data/instance:
uuid: "3c705e0685c14f0f8bfba47d70a3bbec"
format_stamp: "Formatted at 2026-09-26 16:47:33 on dist-test-slave-wt2j"
server_key: "21c6914b9ceee487480bd7fac2b6459c"
server_key_iv: "b0369cb9e787e2237e5f2918cd5a66d3"
server_key_version: "encryptionkey@0"
I20260926 16:47:33.016806 25796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance:
uuid: "3c705e0685c14f0f8bfba47d70a3bbec"
format_stamp: "Formatted at 2026-09-26 16:47:33 on dist-test-slave-wt2j"
server_key: "21c6914b9ceee487480bd7fac2b6459c"
server_key_iv: "b0369cb9e787e2237e5f2918cd5a66d3"
server_key_version: "encryptionkey@0"
I20260926 16:47:33.023396 25796 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 16:47:33.028501 25811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:33.030632 25796 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260926 16:47:33.030903 25796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "3c705e0685c14f0f8bfba47d70a3bbec"
format_stamp: "Formatted at 2026-09-26 16:47:33 on dist-test-slave-wt2j"
server_key: "21c6914b9ceee487480bd7fac2b6459c"
server_key_iv: "b0369cb9e787e2237e5f2918cd5a66d3"
server_key_version: "encryptionkey@0"
I20260926 16:47:33.031178 25796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:47:33.065855 25796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:47:33.071884 25796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:47:33.072343 25796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:47:33.073793 25796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:47:33.076360 25796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:47:33.076467 25796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:33.076586 25796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:47:33.076637 25796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:33.121951 25796 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:44745
I20260926 16:47:33.122013 25923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:44745 every 8 connection(s)
I20260926 16:47:33.123883 25796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:47:33.133841 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 25796
I20260926 16:47:33.134095 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance
I20260926 16:47:33.134831 10171 external_mini_cluster.cc:1428] Setting key 0becbb61b6c4cead6221fdd0e89c6fb6
Sep 26 16:47:33 dist-test-slave-wt2j krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441253, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:47:33.150151 25926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:33.127094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:39391 (local address 127.9.238.254:35375)
0926 16:47:33.127802 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:33.127808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:33.129030 (+ 1222us) server_negotiation.cc:407] Connection header received
0926 16:47:33.131422 (+ 2392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:33.131430 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:33.131598 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:33.131743 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:33.133793 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:33.134810 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:33.137535 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:33.138021 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:33.138193 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:33.142689 (+ 4496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:33.142694 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:33.142697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:33.142733 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:33.145716 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:33.146637 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:33.146644 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:33.146654 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:33.146778 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:33.147262 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:33.147267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:33.147269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:33.147650 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:33.147827 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:33.149335 (+ 1508us) server_negotiation.cc:299] Negotiation successful
0926 16:47:33.149623 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":298,"threads_started":1}
I20260926 16:47:33.151604 25927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:33.127718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.193:39391)
0926 16:47:33.128650 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:33.128708 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:33.131057 (+ 2349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:33.132069 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:33.132080 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:33.132797 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:47:33.133568 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:33.133590 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:33.135204 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:33.135214 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:33.137280 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:33.137303 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:33.138303 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:33.138313 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:33.138594 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:33.139609 (+ 1015us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:47:33.139656 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:47:33.142440 (+ 2784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:47:33.145906 (+ 3466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:33.145923 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:33.145939 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:33.146488 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:33.146896 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:33.146903 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:33.146907 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:33.147125 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:33.147924 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:47:33.147940 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:47:33.148705 (+ 765us) client_negotiation.cc:769] Sending connection context
0926 16:47:33.149163 (+ 458us) client_negotiation.cc:240] Negotiation successful
0926 16:47:33.149751 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":387,"threads_started":1}
I20260926 16:47:33.153906 25924 heartbeater.cc:347] Connected to a master server at 127.9.238.254:35375
I20260926 16:47:33.154368 25924 heartbeater.cc:464] Registering TS with master...
I20260926 16:47:33.155400 25924 heartbeater.cc:511] Master 127.9.238.254:35375 requested a full tablet report, sending...
I20260926 16:47:33.158654 25735 ts_manager.cc:194] Registered new tserver with Master: 3c705e0685c14f0f8bfba47d70a3bbec (127.9.238.193:44745)
I20260926 16:47:33.160802 25735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.193@KRBTEST.COM'} at 127.9.238.193:39391
WARNING: no policy specified for kudu/127.9.238.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
I20260926 16:47:33.203817 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:0
--local_ip_for_outbound_sockets=127.9.238.194
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:35375
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:45093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:47:33.544919 25930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:47:33.545219 25930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:47:33.545317 25930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:47:33.552915 25930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:47:33.553051 25930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:47:33.553172 25930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:47:33.563238 25930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:45093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:35375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:47:33.564726 25930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:47:33.566084 25930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:47:33.581648 25938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:33.581689 25935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:33.581732 25936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:47:33.582844 25930 server_base.cc:1034] running on GCE node
I20260926 16:47:33.583853 25930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:47:33.585201 25930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:47:33.586473 25930 hybrid_clock.cc:648] HybridClock initialized: now 1790441253586402 us; error 56 us; skew 500 ppm
I20260926 16:47:33.586954 25930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:47:33 dist-test-slave-wt2j krb5kdc[25690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441253, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:47:33.591275 25930 init.cc:377] Logged in from keytab as kudu/127.9.238.194@KRBTEST.COM (short username kudu)
I20260926 16:47:33.593320 25930 webserver.cc:460] Webserver started at http://127.9.238.194:41099/ using document root <none> and password file <none>
I20260926 16:47:33.594307 25930 fs_manager.cc:359] Metadata directory not provided
I20260926 16:47:33.594461 25930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:47:33.594874 25930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:47:33.598686 25930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/data/instance:
uuid: "e57ee7c944b64744b83764bb53c16bf6"
format_stamp: "Formatted at 2026-09-26 16:47:33 on dist-test-slave-wt2j"
server_key: "f33fafa982a8b00af00517c6f71a9f84"
server_key_iv: "a6237a5d3f7b619cb2b1e178675251b4"
server_key_version: "encryptionkey@0"
I20260926 16:47:33.599795 25930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance:
uuid: "e57ee7c944b64744b83764bb53c16bf6"
format_stamp: "Formatted at 2026-09-26 16:47:33 on dist-test-slave-wt2j"
server_key: "f33fafa982a8b00af00517c6f71a9f84"
server_key_iv: "a6237a5d3f7b619cb2b1e178675251b4"
server_key_version: "encryptionkey@0"
I20260926 16:47:33.606778 25930 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260926 16:47:33.611837 25945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:33.614056 25930 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:47:33.614336 25930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "e57ee7c944b64744b83764bb53c16bf6"
format_stamp: "Formatted at 2026-09-26 16:47:33 on dist-test-slave-wt2j"
server_key: "f33fafa982a8b00af00517c6f71a9f84"
server_key_iv: "a6237a5d3f7b619cb2b1e178675251b4"
server_key_version: "encryptionkey@0"
I20260926 16:47:33.614605 25930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:47:33.633718 25930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:47:33.639576 25930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:47:33.640017 25930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:47:33.641155 25930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:47:33.642684 25930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:47:33.642788 25930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:33.642941 25930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:47:33.643033 25930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:33.682401 25930 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:42251
I20260926 16:47:33.682425 26057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:42251 every 8 connection(s)
I20260926 16:47:33.684103 25930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:47:33.684911 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 25930
I20260926 16:47:33.685112 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance
I20260926 16:47:33.685732 10171 external_mini_cluster.cc:1428] Setting key d9158583a8829a20da2f3decdd30b5ae
Sep 26 16:47:33 dist-test-slave-wt2j krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441253, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:47:33.709159 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:33.687420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:44081 (local address 127.9.238.254:35375)
0926 16:47:33.687919 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:33.687924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:33.689687 (+ 1763us) server_negotiation.cc:407] Connection header received
0926 16:47:33.692907 (+ 3220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:33.692914 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:33.693020 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:33.693167 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:33.695651 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:33.696547 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:33.698808 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:33.699285 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:33.699436 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:33.703216 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:33.703221 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:33.703225 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:33.703260 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:33.705945 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:33.706774 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:33.706784 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:33.706786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:33.706907 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:33.707342 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:33.707349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:33.707352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:33.707684 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:33.707834 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:33.708525 (+ 691us) server_negotiation.cc:299] Negotiation successful
0926 16:47:33.708756 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":290,"threads_started":1}
I20260926 16:47:33.710170 26061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:33.688339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.194:44081)
0926 16:47:33.689449 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:33.689532 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:33.692494 (+ 2962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:33.693500 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:33.693514 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:33.694355 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:47:33.695295 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:33.695323 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:33.696764 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:33.696773 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:33.698598 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:33.698611 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:33.699569 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:33.699577 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:33.699822 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:33.700654 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:47:33.700678 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:47:33.702965 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:47:33.706154 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:33.706163 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:33.706182 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:33.706644 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:33.707033 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:33.707038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:33.707042 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:33.707229 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:33.707806 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:47:33.707814 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:47:33.708168 (+ 354us) client_negotiation.cc:769] Sending connection context
0926 16:47:33.708437 (+ 269us) client_negotiation.cc:240] Negotiation successful
0926 16:47:33.708996 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":419,"threads_started":1}
I20260926 16:47:33.711694 26058 heartbeater.cc:347] Connected to a master server at 127.9.238.254:35375
I20260926 16:47:33.712013 26058 heartbeater.cc:464] Registering TS with master...
I20260926 16:47:33.712751 26058 heartbeater.cc:511] Master 127.9.238.254:35375 requested a full tablet report, sending...
I20260926 16:47:33.714277 25735 ts_manager.cc:194] Registered new tserver with Master: e57ee7c944b64744b83764bb53c16bf6 (127.9.238.194:42251)
I20260926 16:47:33.715181 25735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.194@KRBTEST.COM'} at 127.9.238.194:44081
WARNING: no policy specified for kudu/127.9.238.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
I20260926 16:47:33.756101 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:0
--local_ip_for_outbound_sockets=127.9.238.195
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:35375
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:45093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:47:34.161586 26064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:47:34.161906 26064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:47:34.162053 26064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 16:47:34.165532 25924 heartbeater.cc:503] Master 127.9.238.254:35375 was elected leader, sending a full tablet report...
W20260926 16:47:34.170307 26064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:47:34.170481 26064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:47:34.170648 26064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:47:34.181526 26064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:45093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:35375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:47:34.182953 26064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:47:34.184442 26064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:47:34.201936 26069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:34.201893 26072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:34.201978 26070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:47:34.203027 26064 server_base.cc:1034] running on GCE node
I20260926 16:47:34.203830 26064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:47:34.204949 26064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:47:34.206176 26064 hybrid_clock.cc:648] HybridClock initialized: now 1790441254206091 us; error 80 us; skew 500 ppm
I20260926 16:47:34.206526 26064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:47:34 dist-test-slave-wt2j krb5kdc[25690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441254, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:47:34.210419 26064 init.cc:377] Logged in from keytab as kudu/127.9.238.195@KRBTEST.COM (short username kudu)
I20260926 16:47:34.212332 26064 webserver.cc:460] Webserver started at http://127.9.238.195:41093/ using document root <none> and password file <none>
I20260926 16:47:34.213136 26064 fs_manager.cc:359] Metadata directory not provided
I20260926 16:47:34.213277 26064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:47:34.213615 26064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:47:34.216813 26064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/data/instance:
uuid: "41c3865003b24affa65fae2b870fc44e"
format_stamp: "Formatted at 2026-09-26 16:47:34 on dist-test-slave-wt2j"
server_key: "b955bf0b553f3596a6d5c8b88ad373f7"
server_key_iv: "8791de76ad05e607fcf0af47da8ab019"
server_key_version: "encryptionkey@0"
I20260926 16:47:34.217634 26064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance:
uuid: "41c3865003b24affa65fae2b870fc44e"
format_stamp: "Formatted at 2026-09-26 16:47:34 on dist-test-slave-wt2j"
server_key: "b955bf0b553f3596a6d5c8b88ad373f7"
server_key_iv: "8791de76ad05e607fcf0af47da8ab019"
server_key_version: "encryptionkey@0"
I20260926 16:47:34.224270 26064 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 16:47:34.228935 26079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:34.230886 26064 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 16:47:34.231133 26064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "41c3865003b24affa65fae2b870fc44e"
format_stamp: "Formatted at 2026-09-26 16:47:34 on dist-test-slave-wt2j"
server_key: "b955bf0b553f3596a6d5c8b88ad373f7"
server_key_iv: "8791de76ad05e607fcf0af47da8ab019"
server_key_version: "encryptionkey@0"
I20260926 16:47:34.231353 26064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:47:34.248224 26064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:47:34.253368 26064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:47:34.253635 26064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:47:34.254685 26064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:47:34.256258 26064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:47:34.256340 26064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:34.256425 26064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:47:34.256486 26064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:34.297032 26064 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:46641
I20260926 16:47:34.297175 26191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:46641 every 8 connection(s)
I20260926 16:47:34.298493 26064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:47:34.307159 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 26064
I20260926 16:47:34.307366 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance
I20260926 16:47:34.307920 10171 external_mini_cluster.cc:1428] Setting key 937f95217f151fbc8cffe292a0f959dd
Sep 26 16:47:34 dist-test-slave-wt2j krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441254, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:47:34.326573 26194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:34.302015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:48407 (local address 127.9.238.254:35375)
0926 16:47:34.302774 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:34.302786 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:34.303622 (+ 836us) server_negotiation.cc:407] Connection header received
0926 16:47:34.306196 (+ 2574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:34.306203 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:34.306551 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:34.306744 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:34.309239 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.310230 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.313092 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.313957 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.314142 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:34.318462 (+ 4320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:34.318469 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:34.318473 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:34.318522 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:34.321972 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:34.323137 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:34.323143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:34.323147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:34.323343 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:34.324081 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:34.324087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:34.324090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:34.324600 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:34.324806 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:34.325896 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0926 16:47:34.326181 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":419,"threads_started":1}
I20260926 16:47:34.327396 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:34.302351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.195:48407)
0926 16:47:34.303382 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:34.303438 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:34.305749 (+ 2311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:34.307098 (+ 1349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:34.307110 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:34.307993 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:47:34.308973 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:34.309010 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.310552 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.310563 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:34.312789 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:34.312819 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.314258 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.314264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:34.314524 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:34.315276 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:47:34.315301 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:47:34.318093 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:47:34.322217 (+ 4124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:34.322226 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:34.322242 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:34.322966 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:34.323529 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:34.323534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:34.323537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:34.323883 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:34.324763 (+ 880us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:47:34.324770 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:47:34.325302 (+ 532us) client_negotiation.cc:769] Sending connection context
0926 16:47:34.325666 (+ 364us) client_negotiation.cc:240] Negotiation successful
0926 16:47:34.326162 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":408,"threads_started":1}
I20260926 16:47:34.329511 26192 heartbeater.cc:347] Connected to a master server at 127.9.238.254:35375
I20260926 16:47:34.329921 26192 heartbeater.cc:464] Registering TS with master...
I20260926 16:47:34.330843 26192 heartbeater.cc:511] Master 127.9.238.254:35375 requested a full tablet report, sending...
I20260926 16:47:34.332844 25735 ts_manager.cc:194] Registered new tserver with Master: 41c3865003b24affa65fae2b870fc44e (127.9.238.195:46641)
I20260926 16:47:34.333832 25735 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.195@KRBTEST.COM'} at 127.9.238.195:48407
I20260926 16:47:34.338954 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:47:34.360525 26194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:34.344740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57586 (local address 127.9.238.254:35375)
0926 16:47:34.345137 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:34.345147 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:34.345593 (+ 446us) server_negotiation.cc:407] Connection header received
0926 16:47:34.345861 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:34.345866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:34.345970 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:34.346085 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:34.347808 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.348614 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.350175 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.350611 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.350757 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:34.353827 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:34.353833 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:34.353836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:34.353880 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:34.357136 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:34.358155 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:34.358161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:34.358165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:34.358327 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:34.358719 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:34.358726 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:34.358730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:34.359152 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:34.359439 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:34.359861 (+ 422us) server_negotiation.cc:299] Negotiation successful
0926 16:47:34.360158 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260926 16:47:34.365725 25735 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:47:34.373204 25735 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:47:34.401834 26205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:34.389482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38344 (local address 127.9.238.195:46641)
0926 16:47:34.390355 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:34.390364 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:34.390931 (+ 567us) server_negotiation.cc:407] Connection header received
0926 16:47:34.391388 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:34.391400 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:34.391804 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:34.392070 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:47:34.393760 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.395495 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.398293 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.399846 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.400023 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:34.400359 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:34.401118 (+ 759us) server_negotiation.cc:299] Negotiation successful
0926 16:47:34.401346 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":431,"threads_started":1}
I20260926 16:47:34.401840 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:34.389452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:46641 (local address 127.0.0.1:38344)
0926 16:47:34.390758 (+ 1306us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:34.390801 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:34.391021 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:34.392562 (+ 1541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:34.392575 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:34.392616 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:47:34.393119 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:34.393138 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.395769 (+ 2631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.395777 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:34.398055 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:34.398075 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.400548 (+ 2473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.400556 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:34.400811 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:34.400878 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 16:47:34.401060 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 16:47:34.401346 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1044,"thread_start_us":863,"threads_started":1}
I20260926 16:47:34.409066 26127 tablet_service.cc:1467] Processing CreateTablet for tablet 1be20ae33fb64e75bbeb4c3ca8d762c3 (DEFAULT_TABLE table=test-table [id=87cc190e7293452381a5a92b5afb9f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:47:34.410795 26127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be20ae33fb64e75bbeb4c3ca8d762c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:47:34.411499 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:34.393106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:44745 (local address 127.0.0.1:33366)
0926 16:47:34.402167 (+ 9061us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:34.402192 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:34.402351 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:34.403747 (+ 1396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:34.403752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:34.403784 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:47:34.403993 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:34.404003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.406256 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.406264 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:34.408351 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:34.408382 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.410482 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.410488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:34.410731 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:34.410790 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 16:47:34.410881 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 16:47:34.411083 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8923,"thread_start_us":11271,"threads_started":1}
I20260926 16:47:34.412539 26209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:34.392696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33366 (local address 127.9.238.193:44745)
0926 16:47:34.400817 (+ 8121us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:34.400829 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:34.402510 (+ 1681us) server_negotiation.cc:407] Connection header received
0926 16:47:34.402666 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:34.402678 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:34.403093 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:34.403380 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:47:34.404154 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.406027 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.408660 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.410259 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.410443 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:34.410676 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:34.411900 (+ 1224us) server_negotiation.cc:299] Negotiation successful
0926 16:47:34.412144 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7419,"thread_start_us":7165,"threads_started":1}
I20260926 16:47:34.413537 26207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:34.391224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:42251 (local address 127.0.0.1:37590)
0926 16:47:34.392155 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:34.392179 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:34.392319 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:34.405942 (+ 13623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:34.405949 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:34.405985 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:47:34.406461 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:34.406477 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.408587 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.408593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:34.410616 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:34.410632 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.412895 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.412900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:34.413037 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:34.413064 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 16:47:34.413120 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 16:47:34.413259 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"thread_start_us":569,"threads_started":1}
I20260926 16:47:34.414700 26208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:34.391688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37590 (local address 127.9.238.194:42251)
0926 16:47:34.405173 (+ 13485us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:34.405184 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:34.405219 (+ 35us) server_negotiation.cc:407] Connection header received
0926 16:47:34.405361 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:34.405373 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:34.405670 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:34.406665 (+ 995us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:47:34.406884 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.408410 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.410825 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.412634 (+ 1809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.413812 (+ 1178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:34.414002 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:34.414101 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 16:47:34.414302 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13270,"thread_start_us":13078,"threads_started":1}
I20260926 16:47:34.416582 25859 tablet_service.cc:1467] Processing CreateTablet for tablet 1be20ae33fb64e75bbeb4c3ca8d762c3 (DEFAULT_TABLE table=test-table [id=87cc190e7293452381a5a92b5afb9f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:47:34.419795 25993 tablet_service.cc:1467] Processing CreateTablet for tablet 1be20ae33fb64e75bbeb4c3ca8d762c3 (DEFAULT_TABLE table=test-table [id=87cc190e7293452381a5a92b5afb9f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:47:34.420691 25859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be20ae33fb64e75bbeb4c3ca8d762c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:47:34.421597 25993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1be20ae33fb64e75bbeb4c3ca8d762c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:47:34.432582 26211 tablet_bootstrap.cc:492] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 41c3865003b24affa65fae2b870fc44e: Bootstrap starting.
I20260926 16:47:34.437711 26211 tablet_bootstrap.cc:654] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 41c3865003b24affa65fae2b870fc44e: Neither blocks nor log segments found. Creating new log.
I20260926 16:47:34.439215 26211 log.cc:824] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 41c3865003b24affa65fae2b870fc44e: Log is configured to *not* fsync() on all Append() calls
I20260926 16:47:34.445566 26213 tablet_bootstrap.cc:492] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P e57ee7c944b64744b83764bb53c16bf6: Bootstrap starting.
I20260926 16:47:34.449105 26211 tablet_bootstrap.cc:492] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 41c3865003b24affa65fae2b870fc44e: No bootstrap required, opened a new log
I20260926 16:47:34.449545 26211 ts_tablet_manager.cc:1397] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 41c3865003b24affa65fae2b870fc44e: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.006s
I20260926 16:47:34.449570 26213 tablet_bootstrap.cc:654] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P e57ee7c944b64744b83764bb53c16bf6: Neither blocks nor log segments found. Creating new log.
I20260926 16:47:34.450994 26213 log.cc:824] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P e57ee7c944b64744b83764bb53c16bf6: Log is configured to *not* fsync() on all Append() calls
I20260926 16:47:34.452382 26212 tablet_bootstrap.cc:492] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec: Bootstrap starting.
I20260926 16:47:34.456216 26213 tablet_bootstrap.cc:492] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P e57ee7c944b64744b83764bb53c16bf6: No bootstrap required, opened a new log
I20260926 16:47:34.456707 26213 ts_tablet_manager.cc:1397] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P e57ee7c944b64744b83764bb53c16bf6: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260926 16:47:34.456777 26212 tablet_bootstrap.cc:654] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec: Neither blocks nor log segments found. Creating new log.
I20260926 16:47:34.458235 26212 log.cc:824] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec: Log is configured to *not* fsync() on all Append() calls
I20260926 16:47:34.460566 26211 raft_consensus.cc:348] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 41c3865003b24affa65fae2b870fc44e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c3865003b24affa65fae2b870fc44e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46641 } } peers { permanent_uuid: "e57ee7c944b64744b83764bb53c16bf6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42251 } } peers { permanent_uuid: "3c705e0685c14f0f8bfba47d70a3bbec" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44745 } }
I20260926 16:47:34.461059 26211 raft_consensus.cc:374] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 41c3865003b24affa65fae2b870fc44e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:47:34.461205 26211 raft_consensus.cc:729] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 41c3865003b24affa65fae2b870fc44e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c3865003b24affa65fae2b870fc44e, State: Initialized, Role: FOLLOWER
I20260926 16:47:34.461948 26211 consensus_queue.cc:260] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 41c3865003b24affa65fae2b870fc44e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c3865003b24affa65fae2b870fc44e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46641 } } peers { permanent_uuid: "e57ee7c944b64744b83764bb53c16bf6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42251 } } peers { permanent_uuid: "3c705e0685c14f0f8bfba47d70a3bbec" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44745 } }
I20260926 16:47:34.462231 26212 tablet_bootstrap.cc:492] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec: No bootstrap required, opened a new log
I20260926 16:47:34.462600 26212 ts_tablet_manager.cc:1397] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260926 16:47:34.463398 26192 heartbeater.cc:503] Master 127.9.238.254:35375 was elected leader, sending a full tablet report...
I20260926 16:47:34.464036 26211 ts_tablet_manager.cc:1428] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 41c3865003b24affa65fae2b870fc44e: Time spent starting tablet: real 0.014s user 0.005s sys 0.009s
I20260926 16:47:34.467428 26213 raft_consensus.cc:348] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P e57ee7c944b64744b83764bb53c16bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c3865003b24affa65fae2b870fc44e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46641 } } peers { permanent_uuid: "e57ee7c944b64744b83764bb53c16bf6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42251 } } peers { permanent_uuid: "3c705e0685c14f0f8bfba47d70a3bbec" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44745 } }
I20260926 16:47:34.468003 26213 raft_consensus.cc:374] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P e57ee7c944b64744b83764bb53c16bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:47:34.468144 26213 raft_consensus.cc:729] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P e57ee7c944b64744b83764bb53c16bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e57ee7c944b64744b83764bb53c16bf6, State: Initialized, Role: FOLLOWER
I20260926 16:47:34.468941 26213 consensus_queue.cc:260] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P e57ee7c944b64744b83764bb53c16bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c3865003b24affa65fae2b870fc44e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46641 } } peers { permanent_uuid: "e57ee7c944b64744b83764bb53c16bf6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42251 } } peers { permanent_uuid: "3c705e0685c14f0f8bfba47d70a3bbec" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44745 } }
I20260926 16:47:34.470510 26058 heartbeater.cc:503] Master 127.9.238.254:35375 was elected leader, sending a full tablet report...
I20260926 16:47:34.471593 26213 ts_tablet_manager.cc:1428] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P e57ee7c944b64744b83764bb53c16bf6: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260926 16:47:34.474705 26212 raft_consensus.cc:348] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c3865003b24affa65fae2b870fc44e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46641 } } peers { permanent_uuid: "e57ee7c944b64744b83764bb53c16bf6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42251 } } peers { permanent_uuid: "3c705e0685c14f0f8bfba47d70a3bbec" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44745 } }
I20260926 16:47:34.475226 26212 raft_consensus.cc:374] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:47:34.475335 26212 raft_consensus.cc:729] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c705e0685c14f0f8bfba47d70a3bbec, State: Initialized, Role: FOLLOWER
I20260926 16:47:34.476115 26212 consensus_queue.cc:260] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c3865003b24affa65fae2b870fc44e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46641 } } peers { permanent_uuid: "e57ee7c944b64744b83764bb53c16bf6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42251 } } peers { permanent_uuid: "3c705e0685c14f0f8bfba47d70a3bbec" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44745 } }
I20260926 16:47:34.478113 26212 ts_tablet_manager.cc:1428] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
W20260926 16:47:34.550510 26193 tablet.cc:2369] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 41c3865003b24affa65fae2b870fc44e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:47:34.627305 25925 tablet.cc:2369] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:47:34.632896 26219 raft_consensus.cc:482] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:47:34.633296 26219 raft_consensus.cc:504] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c3865003b24affa65fae2b870fc44e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46641 } } peers { permanent_uuid: "e57ee7c944b64744b83764bb53c16bf6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42251 } } peers { permanent_uuid: "3c705e0685c14f0f8bfba47d70a3bbec" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44745 } }
I20260926 16:47:34.636026 26219 leader_election.cc:288] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c3865003b24affa65fae2b870fc44e (127.9.238.195:46641), e57ee7c944b64744b83764bb53c16bf6 (127.9.238.194:42251)
I20260926 16:47:34.643191 26205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:34.636347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:52635 (local address 127.9.238.195:46641)
0926 16:47:34.636638 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:34.636650 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:34.637286 (+ 636us) server_negotiation.cc:407] Connection header received
0926 16:47:34.637624 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:34.637631 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:34.637716 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:34.637795 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:47:34.638586 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.639360 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.641336 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.642220 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.642413 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:34.642494 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:34.642780 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 16:47:34.642911 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 16:47:34.643349 26220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:34.636310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:46641 (local address 127.9.238.193:52635)
0926 16:47:34.637174 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:34.637195 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:34.637354 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:34.637988 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:34.637992 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:34.638022 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:47:34.638421 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:34.638431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.639508 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.639515 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:34.641184 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:34.641198 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.642429 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.642436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:34.642613 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:34.642640 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 16:47:34.642742 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 16:47:34.642891 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"spinlock_wait_cycles":58112,"thread_start_us":432,"threads_started":1}
I20260926 16:47:34.644575 26221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:34.637217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:42251 (local address 127.9.238.193:59405)
0926 16:47:34.637959 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:34.637984 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:34.638238 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:34.638705 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:34.638709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:34.638735 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:47:34.639155 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:34.639165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.640354 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.640364 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:34.642151 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:34.642167 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.643746 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.643753 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:34.643920 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:34.643945 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 16:47:34.644065 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 16:47:34.644215 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"spinlock_wait_cycles":1280,"thread_start_us":348,"threads_started":1}
I20260926 16:47:34.644624 26208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:34.637614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:59405 (local address 127.9.238.194:42251)
0926 16:47:34.637904 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:34.637911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:34.638103 (+ 192us) server_negotiation.cc:407] Connection header received
0926 16:47:34.638413 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:34.638420 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:34.638531 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:34.638692 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:47:34.639316 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.640184 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.642342 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.643246 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.643473 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:34.643600 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:34.644129 (+ 529us) server_negotiation.cc:299] Negotiation successful
0926 16:47:34.644337 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 16:47:34.644722 26147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be20ae33fb64e75bbeb4c3ca8d762c3" candidate_uuid: "3c705e0685c14f0f8bfba47d70a3bbec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c3865003b24affa65fae2b870fc44e" is_pre_election: true
I20260926 16:47:34.645438 26147 raft_consensus.cc:2454] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 41c3865003b24affa65fae2b870fc44e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c705e0685c14f0f8bfba47d70a3bbec in term 0.
I20260926 16:47:34.645689 26013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be20ae33fb64e75bbeb4c3ca8d762c3" candidate_uuid: "3c705e0685c14f0f8bfba47d70a3bbec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e57ee7c944b64744b83764bb53c16bf6" is_pre_election: true
I20260926 16:47:34.646318 26013 raft_consensus.cc:2454] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P e57ee7c944b64744b83764bb53c16bf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c705e0685c14f0f8bfba47d70a3bbec in term 0.
I20260926 16:47:34.646317 25812 leader_election.cc:302] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c705e0685c14f0f8bfba47d70a3bbec, 41c3865003b24affa65fae2b870fc44e; no voters:
I20260926 16:47:34.646989 26219 raft_consensus.cc:2781] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:47:34.647267 26219 raft_consensus.cc:482] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:47:34.647418 26219 raft_consensus.cc:3037] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:47:34.650552 26219 raft_consensus.cc:504] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c3865003b24affa65fae2b870fc44e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46641 } } peers { permanent_uuid: "e57ee7c944b64744b83764bb53c16bf6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42251 } } peers { permanent_uuid: "3c705e0685c14f0f8bfba47d70a3bbec" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44745 } }
I20260926 16:47:34.651612 26219 leader_election.cc:288] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec [CANDIDATE]: Term 1 election: Requested vote from peers 41c3865003b24affa65fae2b870fc44e (127.9.238.195:46641), e57ee7c944b64744b83764bb53c16bf6 (127.9.238.194:42251)
I20260926 16:47:34.652419 26147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be20ae33fb64e75bbeb4c3ca8d762c3" candidate_uuid: "3c705e0685c14f0f8bfba47d70a3bbec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c3865003b24affa65fae2b870fc44e"
I20260926 16:47:34.652434 26013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be20ae33fb64e75bbeb4c3ca8d762c3" candidate_uuid: "3c705e0685c14f0f8bfba47d70a3bbec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e57ee7c944b64744b83764bb53c16bf6"
I20260926 16:47:34.652732 26013 raft_consensus.cc:3037] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P e57ee7c944b64744b83764bb53c16bf6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:47:34.652745 26147 raft_consensus.cc:3037] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 41c3865003b24affa65fae2b870fc44e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:47:34.655395 26013 raft_consensus.cc:2454] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P e57ee7c944b64744b83764bb53c16bf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c705e0685c14f0f8bfba47d70a3bbec in term 1.
I20260926 16:47:34.655642 26147 raft_consensus.cc:2454] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 41c3865003b24affa65fae2b870fc44e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c705e0685c14f0f8bfba47d70a3bbec in term 1.
I20260926 16:47:34.655997 25812 leader_election.cc:302] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c705e0685c14f0f8bfba47d70a3bbec, e57ee7c944b64744b83764bb53c16bf6; no voters:
I20260926 16:47:34.656512 26219 raft_consensus.cc:2781] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:47:34.657204 26219 raft_consensus.cc:686] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec [term 1 LEADER]: Becoming Leader. State: Replica: 3c705e0685c14f0f8bfba47d70a3bbec, State: Running, Role: LEADER
I20260926 16:47:34.657866 26219 consensus_queue.cc:237] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c3865003b24affa65fae2b870fc44e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46641 } } peers { permanent_uuid: "e57ee7c944b64744b83764bb53c16bf6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42251 } } peers { permanent_uuid: "3c705e0685c14f0f8bfba47d70a3bbec" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44745 } }
I20260926 16:47:34.664337 25735 catalog_manager.cc:5495] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c705e0685c14f0f8bfba47d70a3bbec (127.9.238.193). New cstate: current_term: 1 leader_uuid: "3c705e0685c14f0f8bfba47d70a3bbec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c3865003b24affa65fae2b870fc44e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e57ee7c944b64744b83764bb53c16bf6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c705e0685c14f0f8bfba47d70a3bbec" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44745 } health_report { overall_health: HEALTHY } } }
W20260926 16:47:34.687737 26059 tablet.cc:2369] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P e57ee7c944b64744b83764bb53c16bf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:47:34.710575 26209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:34.703490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33378 (local address 127.9.238.193:44745)
0926 16:47:34.703907 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:34.703912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:34.703993 (+ 81us) server_negotiation.cc:407] Connection header received
0926 16:47:34.704241 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:34.704246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:34.704340 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:34.704492 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:47:34.705153 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.705953 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.707760 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.708219 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.708347 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:34.709075 (+ 728us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:47:34.709773 (+ 698us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:47:34.709935 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:34.710073 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 16:47:34.710165 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260926 16:47:34.721405 26013 raft_consensus.cc:1261] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P e57ee7c944b64744b83764bb53c16bf6 [term 1 FOLLOWER]: Refusing update from remote peer 3c705e0685c14f0f8bfba47d70a3bbec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:47:34.721524 26147 raft_consensus.cc:1261] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 41c3865003b24affa65fae2b870fc44e [term 1 FOLLOWER]: Refusing update from remote peer 3c705e0685c14f0f8bfba47d70a3bbec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:47:34.722724 26222 consensus_queue.cc:1035] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec [LEADER]: Connected to new peer: Peer: permanent_uuid: "e57ee7c944b64744b83764bb53c16bf6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:47:34.723230 26219 consensus_queue.cc:1035] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c3865003b24affa65fae2b870fc44e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:47:34.741006 26227 mvcc.cc:205] Tried to move back new op lower bound from 7333647379325825024 to 7333647379083902976. Current Snapshot: MvccSnapshot[applied={T|T < 7333647379325825024}]
I20260926 16:47:34.753468 25734 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:47:34.754030 25734 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:47:34.758973 25734 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4: Sending DeleteTablet for 3 replicas of tablet 1be20ae33fb64e75bbeb4c3ca8d762c3
I20260926 16:47:34.760522 26127 tablet_service.cc:1514] Processing DeleteTablet for tablet 1be20ae33fb64e75bbeb4c3ca8d762c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:47:34 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:38344
I20260926 16:47:34.760797 25993 tablet_service.cc:1514] Processing DeleteTablet for tablet 1be20ae33fb64e75bbeb4c3ca8d762c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:47:34 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:37590
I20260926 16:47:34.761271 25859 tablet_service.cc:1514] Processing DeleteTablet for tablet 1be20ae33fb64e75bbeb4c3ca8d762c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:47:34 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:33366
I20260926 16:47:34.761672 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 25796
I20260926 16:47:34.793368 26236 tablet_replica.cc:330] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 41c3865003b24affa65fae2b870fc44e: stopping tablet replica
I20260926 16:47:34.794364 26238 tablet_replica.cc:330] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec: stopping tablet replica
I20260926 16:47:34.795176 26238 raft_consensus.cc:2229] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:47:34.795219 26236 raft_consensus.cc:2229] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 41c3865003b24affa65fae2b870fc44e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:47:34.796098 26236 raft_consensus.cc:2258] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 41c3865003b24affa65fae2b870fc44e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:47:34.796464 26238 raft_consensus.cc:2258] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:47:34.800405 26237 tablet_replica.cc:330] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P e57ee7c944b64744b83764bb53c16bf6: stopping tablet replica
I20260926 16:47:34.801092 26237 raft_consensus.cc:2229] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P e57ee7c944b64744b83764bb53c16bf6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:47:34.801806 26237 raft_consensus.cc:2258] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P e57ee7c944b64744b83764bb53c16bf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:47:34.801800 26238 ts_tablet_manager.cc:1905] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:47:34.804634 26237 ts_tablet_manager.cc:1905] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P e57ee7c944b64744b83764bb53c16bf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:47:34.805243 26236 ts_tablet_manager.cc:1905] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 41c3865003b24affa65fae2b870fc44e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:47:34.810194 26238 ts_tablet_manager.cc:1918] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:47:34.810433 26238 log.cc:1196] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/wal/wals/1be20ae33fb64e75bbeb4c3ca8d762c3
I20260926 16:47:34.811054 26238 ts_tablet_manager.cc:1939] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 3c705e0685c14f0f8bfba47d70a3bbec: Deleting consensus metadata
Sep 26 16:47:34 dist-test-slave-wt2j krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.193@KRBTEST.COM
I20260926 16:47:34.813230 25719 catalog_manager.cc:4841] TS 3c705e0685c14f0f8bfba47d70a3bbec (127.9.238.193:44745): tablet 1be20ae33fb64e75bbeb4c3ca8d762c3 (table test-table [id=87cc190e7293452381a5a92b5afb9f1a]) successfully deleted
I20260926 16:47:34.813663 26237 ts_tablet_manager.cc:1918] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P e57ee7c944b64744b83764bb53c16bf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:47:34.814057 26237 log.cc:1196] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P e57ee7c944b64744b83764bb53c16bf6: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/wal/wals/1be20ae33fb64e75bbeb4c3ca8d762c3
I20260926 16:47:34.814756 26237 ts_tablet_manager.cc:1939] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P e57ee7c944b64744b83764bb53c16bf6: Deleting consensus metadata
I20260926 16:47:34.816969 26236 ts_tablet_manager.cc:1918] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 41c3865003b24affa65fae2b870fc44e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:47:34.817178 26236 log.cc:1196] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 41c3865003b24affa65fae2b870fc44e: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/wal/wals/1be20ae33fb64e75bbeb4c3ca8d762c3
I20260926 16:47:34.817715 26236 ts_tablet_manager.cc:1939] T 1be20ae33fb64e75bbeb4c3ca8d762c3 P 41c3865003b24affa65fae2b870fc44e: Deleting consensus metadata
I20260926 16:47:34.817719 25719 catalog_manager.cc:4841] TS e57ee7c944b64744b83764bb53c16bf6 (127.9.238.194:42251): tablet 1be20ae33fb64e75bbeb4c3ca8d762c3 (table test-table [id=87cc190e7293452381a5a92b5afb9f1a]) successfully deleted
I20260926 16:47:34.819778 25719 catalog_manager.cc:4841] TS 41c3865003b24affa65fae2b870fc44e (127.9.238.195:46641): tablet 1be20ae33fb64e75bbeb4c3ca8d762c3 (table test-table [id=87cc190e7293452381a5a92b5afb9f1a]) successfully deleted
I20260926 16:47:34.821704 26209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:34.763520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33392 (local address 127.9.238.193:44745)
0926 16:47:34.763819 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:34.763828 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:34.785683 (+ 21855us) server_negotiation.cc:407] Connection header received
0926 16:47:34.786383 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:34.786394 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:34.786534 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:34.786653 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:34.790423 (+ 3770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.792452 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.799781 (+ 7329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.800657 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.800828 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:34.813031 (+ 12203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:34.813049 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:34.813069 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:34.813127 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:34.817628 (+ 4501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:34.818743 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:34.818756 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:34.818760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:34.819053 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:34.819797 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:34.819810 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:34.819815 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:34.820322 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:34.820641 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:34.821160 (+ 519us) server_negotiation.cc:299] Negotiation successful
0926 16:47:34.821375 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260926 16:47:34.822348 25919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:47:34.939921 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 25796
I20260926 16:47:34.963274 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 25930
Sep 26 16:47:34 dist-test-slave-wt2j krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.194@KRBTEST.COM
I20260926 16:47:34.981376 26208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:34.964267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37604 (local address 127.9.238.194:42251)
0926 16:47:34.964545 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:34.964552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:34.964576 (+ 24us) server_negotiation.cc:407] Connection header received
0926 16:47:34.964713 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:34.964719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:34.964804 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:34.964921 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:34.966385 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.967232 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.968819 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:34.969375 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:34.969518 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:34.974522 (+ 5004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:34.974541 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:34.974559 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:34.974604 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:34.977929 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:34.978780 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:34.978788 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:34.978791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:34.978932 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:34.979322 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:34.979328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:34.979332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:34.980094 (+ 762us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:34.980406 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:34.980768 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 16:47:34.980993 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260926 16:47:34.982333 26053 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:47:35.081949 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 25930
I20260926 16:47:35.101609 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 26064
Sep 26 16:47:35 dist-test-slave-wt2j krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.195@KRBTEST.COM
I20260926 16:47:35.126719 26205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:35.102756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38352 (local address 127.9.238.195:46641)
0926 16:47:35.102981 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:35.102985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:35.103010 (+ 25us) server_negotiation.cc:407] Connection header received
0926 16:47:35.103226 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:35.103230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:35.103309 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:35.103431 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:35.105311 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:35.106096 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:35.107818 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:35.108267 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:35.108412 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:35.119834 (+ 11422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:35.119860 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:35.119879 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:35.119937 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:35.123179 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:35.124400 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:35.124408 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:35.124411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:35.124588 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:35.125129 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:35.125133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:35.125136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:35.125529 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:35.125784 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:35.126255 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 16:47:35.126446 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 16:47:35.127835 26187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:47:35.229861 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 26064
I20260926 16:47:35.250664 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 25703
I20260926 16:47:35.252302 25765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:47:35.378165 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 25703
I20260926 16:47:55.405824 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:35375
--webserver_interface=127.9.238.254
--webserver_port=39433
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:45093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:35375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:47:55.767311 26244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:47:55.767728 26244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:47:55.767907 26244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:47:55.775770 26244 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 16:47:55.775893 26244 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 16:47:55.775962 26244 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 16:47:55.776057 26244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:47:55.776161 26244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:47:55.776273 26244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:47:55.776363 26244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:47:55.776463 26244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:47:55.787010 26244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:45093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:35375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:35375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=39433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:47:55.788473 26244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:47:55.789858 26244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:47:55.816963 26253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:55.817039 26250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:55.817070 26251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:47:55.818614 26244 server_base.cc:1034] running on GCE node
I20260926 16:47:55.819752 26244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:47:55.821768 26244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:47:55.823163 26244 hybrid_clock.cc:648] HybridClock initialized: now 1790441275823057 us; error 68 us; skew 500 ppm
I20260926 16:47:55.823684 26244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:47:55 dist-test-slave-wt2j krb5kdc[25690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441275, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:47:55.827862 26244 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:47:55.829878 26244 webserver.cc:460] Webserver started at http://127.9.238.254:39433/ using document root <none> and password file <none>
I20260926 16:47:55.830740 26244 fs_manager.cc:359] Metadata directory not provided
I20260926 16:47:55.830896 26244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:47:55.837911 26244 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 16:47:55.842060 26260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:55.844018 26244 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 16:47:55.844350 26244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "367b3ecfd4f3444999c5e41fa619c7c4"
format_stamp: "Formatted at 2026-09-26 16:47:32 on dist-test-slave-wt2j"
server_key: "9e3f04b9ec02587a3063bbf571fe0483"
server_key_iv: "14edfba153f0ac42caa1f6136a40b942"
server_key_version: "encryptionkey@0"
I20260926 16:47:55.845597 26244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:47:55.866719 26244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:47:55.871799 26244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:47:55.872145 26244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:47:55.894726 26244 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:35375
I20260926 16:47:55.894753 26311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:35375 every 8 connection(s)
I20260926 16:47:55.896435 26244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:47:55.898178 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 26244
I20260926 16:47:55.899434 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:44745
--local_ip_for_outbound_sockets=127.9.238.193
--tserver_master_addrs=127.9.238.254:35375
--webserver_port=37245
--webserver_interface=127.9.238.193
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:45093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:47:55.909101 26312 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:47:55.914309 26312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4: Bootstrap starting.
I20260926 16:47:55.939411 26312 log.cc:824] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4: Log is configured to *not* fsync() on all Append() calls
I20260926 16:47:55.954793 26312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:47:55.955483 26312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4: Bootstrap complete.
I20260926 16:47:55.966245 26312 raft_consensus.cc:348] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367b3ecfd4f3444999c5e41fa619c7c4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35375 } }
I20260926 16:47:55.967262 26312 raft_consensus.cc:729] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 367b3ecfd4f3444999c5e41fa619c7c4, State: Initialized, Role: FOLLOWER
I20260926 16:47:55.968164 26312 consensus_queue.cc:260] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367b3ecfd4f3444999c5e41fa619c7c4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35375 } }
I20260926 16:47:55.968458 26312 raft_consensus.cc:388] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:47:55.968591 26312 raft_consensus.cc:482] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:47:55.968811 26312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [term 1 FOLLOWER]: Advancing to term 2
I20260926 16:47:55.973650 26312 raft_consensus.cc:504] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367b3ecfd4f3444999c5e41fa619c7c4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35375 } }
I20260926 16:47:55.974424 26312 leader_election.cc:302] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 367b3ecfd4f3444999c5e41fa619c7c4; no voters:
I20260926 16:47:55.975232 26312 leader_election.cc:288] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 16:47:55.975418 26316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 16:47:55.977246 26316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [term 2 LEADER]: Becoming Leader. State: Replica: 367b3ecfd4f3444999c5e41fa619c7c4, State: Running, Role: LEADER
I20260926 16:47:55.978093 26316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367b3ecfd4f3444999c5e41fa619c7c4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35375 } }
I20260926 16:47:55.978500 26312 sys_catalog.cc:564] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:47:55.981123 26318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 367b3ecfd4f3444999c5e41fa619c7c4. Latest consensus state: current_term: 2 leader_uuid: "367b3ecfd4f3444999c5e41fa619c7c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367b3ecfd4f3444999c5e41fa619c7c4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35375 } } }
I20260926 16:47:55.981259 26317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "367b3ecfd4f3444999c5e41fa619c7c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367b3ecfd4f3444999c5e41fa619c7c4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35375 } } }
I20260926 16:47:55.981653 26318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:47:55.981752 26317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:47:55.986629 26323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:47:55.998045 26323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:47:56.001400 26323 catalog_manager.cc:1259] Loaded cluster ID: 6fc673e8d4434f8997712dcaed43249a
I20260926 16:47:56.001698 26323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:47:56.008594 26323 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:47:56.010238 26333 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 16:47:56.011229 26323 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4: Loaded TSK: 0
I20260926 16:47:56.048998 26323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4: Generated new TSK 1
I20260926 16:47:56.049917 26323 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 16:47:56.344313 26314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:47:56.344748 26314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:47:56.344929 26314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:47:56.353652 26314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:47:56.353852 26314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:47:56.354048 26314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:47:56.365669 26314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:45093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:44745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=37245
--tserver_master_addrs=127.9.238.254:35375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:47:56.367496 26314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:47:56.369189 26314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:47:56.388597 26342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:56.388754 26339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:56.389312 26340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:47:56.389810 26314 server_base.cc:1034] running on GCE node
I20260926 16:47:56.390565 26314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:47:56.391934 26314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:47:56.393235 26314 hybrid_clock.cc:648] HybridClock initialized: now 1790441276393151 us; error 76 us; skew 500 ppm
I20260926 16:47:56.393731 26314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:47:56 dist-test-slave-wt2j krb5kdc[25690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441276, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:47:56.397776 26314 init.cc:377] Logged in from keytab as kudu/127.9.238.193@KRBTEST.COM (short username kudu)
I20260926 16:47:56.399832 26314 webserver.cc:460] Webserver started at http://127.9.238.193:37245/ using document root <none> and password file <none>
I20260926 16:47:56.400728 26314 fs_manager.cc:359] Metadata directory not provided
I20260926 16:47:56.400903 26314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:47:56.408113 26314 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.003s
I20260926 16:47:56.412160 26349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:56.414271 26314 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:47:56.414570 26314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "3c705e0685c14f0f8bfba47d70a3bbec"
format_stamp: "Formatted at 2026-09-26 16:47:33 on dist-test-slave-wt2j"
server_key: "21c6914b9ceee487480bd7fac2b6459c"
server_key_iv: "b0369cb9e787e2237e5f2918cd5a66d3"
server_key_version: "encryptionkey@0"
I20260926 16:47:56.416122 26314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:47:56.433645 26314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:47:56.439220 26314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:47:56.439633 26314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:47:56.440977 26314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:47:56.442998 26314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:47:56.443104 26314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:56.443228 26314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:47:56.443336 26314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:56.485581 26314 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:44745
I20260926 16:47:56.485630 26461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:44745 every 8 connection(s)
I20260926 16:47:56.487695 26314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:47:56.497119 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 26314
I20260926 16:47:56.497923 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:42251
--local_ip_for_outbound_sockets=127.9.238.194
--tserver_master_addrs=127.9.238.254:35375
--webserver_port=41099
--webserver_interface=127.9.238.194
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:45093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:47:56 dist-test-slave-wt2j krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441276, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:47:56.513903 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:56.492048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.193:38239)
0926 16:47:56.492967 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:56.493019 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:56.494971 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:56.496536 (+ 1565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:56.496547 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:56.497284 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:47:56.497865 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:56.497883 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:56.499622 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:56.499626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:56.500908 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:56.500921 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:56.501998 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:56.502006 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:56.502422 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:56.503212 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:47:56.503240 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:47:56.505618 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:47:56.509266 (+ 3648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:56.509275 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:56.509290 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:56.509766 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:56.510313 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:56.510318 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:56.510322 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:56.510515 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:56.511285 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:47:56.511295 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:47:56.511712 (+ 417us) client_negotiation.cc:769] Sending connection context
0926 16:47:56.512001 (+ 289us) client_negotiation.cc:240] Negotiation successful
0926 16:47:56.512570 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":340,"threads_started":1}
I20260926 16:47:56.513891 26464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:56.491628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:38239 (local address 127.9.238.254:35375)
0926 16:47:56.492309 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:56.492326 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:56.493191 (+ 865us) server_negotiation.cc:407] Connection header received
0926 16:47:56.495626 (+ 2435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:56.495640 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:56.496002 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:56.496378 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:56.498074 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:56.499382 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:56.501071 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:56.501723 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:56.501949 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:56.505937 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:56.505945 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:56.505961 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:56.506003 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:56.509070 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:56.509952 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:56.509960 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:56.509965 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:56.510186 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:56.510639 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:56.510648 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:56.510650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:56.511145 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:56.511355 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:56.512203 (+ 848us) server_negotiation.cc:299] Negotiation successful
0926 16:47:56.512579 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":284,"threads_started":1}
I20260926 16:47:56.516732 26462 heartbeater.cc:347] Connected to a master server at 127.9.238.254:35375
I20260926 16:47:56.517056 26462 heartbeater.cc:464] Registering TS with master...
I20260926 16:47:56.517839 26462 heartbeater.cc:511] Master 127.9.238.254:35375 requested a full tablet report, sending...
I20260926 16:47:56.520143 26277 ts_manager.cc:194] Registered new tserver with Master: 3c705e0685c14f0f8bfba47d70a3bbec (127.9.238.193:44745)
I20260926 16:47:56.521836 26277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.193@KRBTEST.COM'} at 127.9.238.193:38239
W20260926 16:47:56.862367 26466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:47:56.862733 26466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:47:56.862903 26466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:47:56.870769 26466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:47:56.870949 26466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:47:56.871145 26466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:47:56.881886 26466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:45093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:42251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=41099
--tserver_master_addrs=127.9.238.254:35375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:47:56.883379 26466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:47:56.884845 26466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:47:56.902596 26471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:56.902899 26474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:56.902647 26472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:47:56.904385 26466 server_base.cc:1034] running on GCE node
I20260926 16:47:56.905274 26466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:47:56.906642 26466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:47:56.907981 26466 hybrid_clock.cc:648] HybridClock initialized: now 1790441276907911 us; error 71 us; skew 500 ppm
I20260926 16:47:56.908433 26466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:47:56 dist-test-slave-wt2j krb5kdc[25690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441276, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:47:56.912711 26466 init.cc:377] Logged in from keytab as kudu/127.9.238.194@KRBTEST.COM (short username kudu)
I20260926 16:47:56.914816 26466 webserver.cc:460] Webserver started at http://127.9.238.194:41099/ using document root <none> and password file <none>
I20260926 16:47:56.915827 26466 fs_manager.cc:359] Metadata directory not provided
I20260926 16:47:56.916006 26466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:47:56.923712 26466 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.005s
I20260926 16:47:56.927676 26481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:56.929891 26466 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:47:56.930120 26466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "e57ee7c944b64744b83764bb53c16bf6"
format_stamp: "Formatted at 2026-09-26 16:47:33 on dist-test-slave-wt2j"
server_key: "f33fafa982a8b00af00517c6f71a9f84"
server_key_iv: "a6237a5d3f7b619cb2b1e178675251b4"
server_key_version: "encryptionkey@0"
I20260926 16:47:56.931402 26466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:47:56.961263 26466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:47:56.966915 26466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:47:56.967238 26466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:47:56.968546 26466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:47:56.970247 26466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:47:56.970355 26466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:56.970438 26466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:47:56.970498 26466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:57.026739 26466 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:42251
I20260926 16:47:57.026805 26593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:42251 every 8 connection(s)
I20260926 16:47:57.028769 26466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:47:57.030814 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 26466
I20260926 16:47:57.031798 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:46641
--local_ip_for_outbound_sockets=127.9.238.195
--tserver_master_addrs=127.9.238.254:35375
--webserver_port=41093
--webserver_interface=127.9.238.195
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:45093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:47:57 dist-test-slave-wt2j krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441276, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:47:57.058923 26596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:57.032221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:46129 (local address 127.9.238.254:35375)
0926 16:47:57.032755 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:57.032760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:57.034334 (+ 1574us) server_negotiation.cc:407] Connection header received
0926 16:47:57.037353 (+ 3019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:57.037367 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:57.037512 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:57.037701 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:57.040006 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.042688 (+ 2682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.044639 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.045510 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.045691 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:57.050699 (+ 5008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:57.050705 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:57.050710 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:57.050787 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:57.053734 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:57.055478 (+ 1744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:57.055486 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:57.055489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:57.055724 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:57.056486 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:57.056490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:57.056492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:57.056833 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:57.056964 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:57.058389 (+ 1425us) server_negotiation.cc:299] Negotiation successful
0926 16:47:57.058648 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":272,"threads_started":1}
I20260926 16:47:57.061467 26597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:57.033117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.194:46129)
0926 16:47:57.034114 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:57.034167 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:57.036868 (+ 2701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:57.037996 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:57.038011 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:57.038830 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:47:57.039735 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:57.039764 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.042939 (+ 3175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.042944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:57.044427 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:57.044443 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.045793 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.045798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:57.046060 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:57.046927 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:47:57.046951 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:47:57.050304 (+ 3353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:47:57.053983 (+ 3679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:57.054000 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:57.054032 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:57.055230 (+ 1198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:57.055892 (+ 662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:57.055903 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:57.055911 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:57.056295 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:57.057005 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:47:57.057020 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:47:57.057755 (+ 735us) client_negotiation.cc:769] Sending connection context
0926 16:47:57.058258 (+ 503us) client_negotiation.cc:240] Negotiation successful
0926 16:47:57.059179 (+ 921us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":400,"threads_started":1}
I20260926 16:47:57.063751 26594 heartbeater.cc:347] Connected to a master server at 127.9.238.254:35375
I20260926 16:47:57.064287 26594 heartbeater.cc:464] Registering TS with master...
I20260926 16:47:57.065151 26594 heartbeater.cc:511] Master 127.9.238.254:35375 requested a full tablet report, sending...
I20260926 16:47:57.067006 26277 ts_manager.cc:194] Registered new tserver with Master: e57ee7c944b64744b83764bb53c16bf6 (127.9.238.194:42251)
I20260926 16:47:57.067956 26277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.194@KRBTEST.COM'} at 127.9.238.194:46129
W20260926 16:47:57.463352 26598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:47:57.463858 26598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:47:57.464071 26598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:47:57.473459 26598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:47:57.473680 26598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:47:57.473887 26598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:47:57.486414 26598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:45093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:46641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=41093
--tserver_master_addrs=127.9.238.254:35375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:47:57.488396 26598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:47:57.490103 26598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:47:57.507535 26604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:57.507759 26606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:47:57.507773 26603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:47:57.509721 26598 server_base.cc:1034] running on GCE node
I20260926 16:47:57.510814 26598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:47:57.512480 26598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:47:57.513893 26598 hybrid_clock.cc:648] HybridClock initialized: now 1790441277513821 us; error 49 us; skew 500 ppm
I20260926 16:47:57.514454 26598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:47:57 dist-test-slave-wt2j krb5kdc[25690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441277, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:47:57.519052 26598 init.cc:377] Logged in from keytab as kudu/127.9.238.195@KRBTEST.COM (short username kudu)
I20260926 16:47:57.521549 26598 webserver.cc:460] Webserver started at http://127.9.238.195:41093/ using document root <none> and password file <none>
I20260926 16:47:57.522635 26598 fs_manager.cc:359] Metadata directory not provided
I20260926 16:47:57.522835 26598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:47:57.525254 26462 heartbeater.cc:503] Master 127.9.238.254:35375 was elected leader, sending a full tablet report...
I20260926 16:47:57.531711 26598 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.002s
I20260926 16:47:57.536558 26613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:57.539109 26598 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20260926 16:47:57.539449 26598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "41c3865003b24affa65fae2b870fc44e"
format_stamp: "Formatted at 2026-09-26 16:47:34 on dist-test-slave-wt2j"
server_key: "b955bf0b553f3596a6d5c8b88ad373f7"
server_key_iv: "8791de76ad05e607fcf0af47da8ab019"
server_key_version: "encryptionkey@0"
I20260926 16:47:57.540959 26598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:47:57.566818 26598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:47:57.572466 26598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:47:57.572798 26598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:47:57.574023 26598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:47:57.575781 26598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:47:57.575886 26598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:57.576253 26598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:47:57.576362 26598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:47:57.618058 26598 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:46641
I20260926 16:47:57.618109 26725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:46641 every 8 connection(s)
I20260926 16:47:57.619656 26598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:47:57.622303 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 26598
Sep 26 16:47:57 dist-test-slave-wt2j krb5kdc[25690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441277, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:47:57.658885 26729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:57.623629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41274 (local address 127.9.238.254:35375)
0926 16:47:57.639110 (+ 15481us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:57.639120 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:57.639427 (+ 307us) server_negotiation.cc:407] Connection header received
0926 16:47:57.639827 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:57.639834 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:57.639967 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:57.640169 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:57.642329 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.643846 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.645689 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.646678 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.646865 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:57.650803 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:57.650811 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:57.650817 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:57.650921 (+ 104us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:57.654589 (+ 3668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:57.655702 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:57.655710 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:57.655716 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:57.655931 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:57.656472 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:57.656477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:57.656480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:57.656950 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:57.657276 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:57.657870 (+ 594us) server_negotiation.cc:299] Negotiation successful
0926 16:47:57.658485 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15242,"thread_start_us":16363,"threads_started":1}
I20260926 16:47:57.664779 26728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:57.623677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:35853 (local address 127.9.238.254:35375)
0926 16:47:57.640006 (+ 16329us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:57.640012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:57.640998 (+ 986us) server_negotiation.cc:407] Connection header received
0926 16:47:57.644390 (+ 3392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:57.644396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:57.644491 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:57.644682 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:57.647574 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.648869 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.651502 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.652238 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.652422 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:57.657546 (+ 5124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:57.657551 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:57.657554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:57.657585 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:57.661064 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:57.662107 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:57.662113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:57.662118 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:57.662308 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:57.662846 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:57.662849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:57.662851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:57.663176 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:57.663304 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:57.664183 (+ 879us) server_negotiation.cc:299] Negotiation successful
0926 16:47:57.664431 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16200,"spinlock_wait_cycles":18816,"thread_start_us":15482,"threads_started":1}
I20260926 16:47:57.665800 26731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:57.624934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.195:35853)
0926 16:47:57.640748 (+ 15814us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:57.640834 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:57.643973 (+ 3139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:57.645071 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:57.645083 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:57.646109 (+ 1026us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:47:57.647292 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:57.647322 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.649110 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.649115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:57.651233 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:57.651256 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.652641 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.652650 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:57.652962 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:57.653784 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:47:57.653817 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:47:57.657210 (+ 3393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:47:57.661268 (+ 4058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:57.661277 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:57.661303 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:57.661890 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:57.662462 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:47:57.662466 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:47:57.662470 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:47:57.662733 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:47:57.663319 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:47:57.663325 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:47:57.663821 (+ 496us) client_negotiation.cc:769] Sending connection context
0926 16:47:57.664089 (+ 268us) client_negotiation.cc:240] Negotiation successful
0926 16:47:57.664521 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15284,"thread_start_us":14899,"threads_started":1}
I20260926 16:47:57.667642 26726 heartbeater.cc:347] Connected to a master server at 127.9.238.254:35375
I20260926 16:47:57.668109 26726 heartbeater.cc:464] Registering TS with master...
I20260926 16:47:57.669081 26726 heartbeater.cc:511] Master 127.9.238.254:35375 requested a full tablet report, sending...
I20260926 16:47:57.671233 26277 ts_manager.cc:194] Registered new tserver with Master: 41c3865003b24affa65fae2b870fc44e (127.9.238.195:46641)
I20260926 16:47:57.672216 26277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.195@KRBTEST.COM'} at 127.9.238.195:35853
I20260926 16:47:57.675823 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:47:57.684329 26728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:57.676876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41290 (local address 127.9.238.254:35375)
0926 16:47:57.677196 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:57.677206 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:57.677803 (+ 597us) server_negotiation.cc:407] Connection header received
0926 16:47:57.678070 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:57.678075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:57.678256 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:57.678457 (+ 201us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:47:57.678910 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.680011 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.681633 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.682166 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.682302 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:57.683041 (+ 739us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:47:57.683376 (+ 335us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 16:47:57.683894 (+ 518us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41290: token expired
Metrics: {"server-negotiator.queue_time_us":104}
W20260926 16:47:57.684521 26728 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41290: token expired
W20260926 16:47:57.685184 26196 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.9.238.254:35375)
I20260926 16:47:57.701591 26728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:57.685788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41300 (local address 127.9.238.254:35375)
0926 16:47:57.686014 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:57.686019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:57.686038 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:47:57.686352 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:57.686356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:57.686477 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:57.686639 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:57.688473 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.689468 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.690943 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.691509 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.691767 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:57.694689 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:57.694695 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:57.694699 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:57.694749 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:57.698114 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:57.699035 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:57.699040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:57.699043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:57.699207 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:57.699716 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:57.699723 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:57.699728 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:57.700319 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:57.700644 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:57.700918 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 16:47:57.701152 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 16:47:57.706038 26277 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41300:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:47:57.714622 26277 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:47:57.745499 26737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:57.731875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55334 (local address 127.9.238.193:44745)
0926 16:47:57.732795 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:57.732802 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:57.733144 (+ 342us) server_negotiation.cc:407] Connection header received
0926 16:47:57.733527 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:57.733537 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:57.733870 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:57.735167 (+ 1297us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:47:57.736372 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.738575 (+ 2203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.742519 (+ 3944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.744197 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.744345 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:57.744610 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:57.744912 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 16:47:57.745097 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":627,"threads_started":1}
I20260926 16:47:57.745555 26738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:57.731854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:44745 (local address 127.0.0.1:55334)
0926 16:47:57.732905 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:57.732961 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:57.733220 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:57.734263 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:57.734274 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:57.734332 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:47:57.734915 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:57.734933 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.738791 (+ 3858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.738800 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:57.742254 (+ 3454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:57.742277 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.744429 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.744436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:57.744662 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:57.744714 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 16:47:57.744866 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 16:47:57.745085 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":567,"threads_started":1}
I20260926 16:47:57.747474 26739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:57.733515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:46641 (local address 127.0.0.1:57336)
0926 16:47:57.734147 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:57.734168 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:57.734285 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:57.735886 (+ 1601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:57.735894 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:57.735934 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:47:57.736502 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:57.736519 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.740423 (+ 3904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.740436 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:57.743839 (+ 3403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:57.743864 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.746564 (+ 2700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.746571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:57.746759 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:57.746795 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 16:47:57.746891 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 16:47:57.747056 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":1,"thread_start_us":352,"threads_started":1}
I20260926 16:47:57.748023 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:57.733986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57336 (local address 127.9.238.195:46641)
0926 16:47:57.734986 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:57.735000 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:57.735049 (+ 49us) server_negotiation.cc:407] Connection header received
0926 16:47:57.735197 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:57.735209 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:57.735600 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:57.736748 (+ 1148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:47:57.737180 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.740066 (+ 2886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.744087 (+ 4021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.746303 (+ 2216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.746472 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:57.746782 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:57.746981 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 16:47:57.747196 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":746,"threads_started":1}
I20260926 16:47:57.751375 26397 tablet_service.cc:1467] Processing CreateTablet for tablet ec47fa7509424950b44b1bbad9682dff (DEFAULT_TABLE table=test-table [id=40a0e827721f4b2891de2dcd8aadcb95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:47:57.752722 26661 tablet_service.cc:1467] Processing CreateTablet for tablet ec47fa7509424950b44b1bbad9682dff (DEFAULT_TABLE table=test-table [id=40a0e827721f4b2891de2dcd8aadcb95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:47:57.753579 26397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec47fa7509424950b44b1bbad9682dff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:47:57.754396 26661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec47fa7509424950b44b1bbad9682dff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:47:57.755260 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:57.734577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41380 (local address 127.9.238.194:42251)
0926 16:47:57.742722 (+ 8145us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:57.742735 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:57.744092 (+ 1357us) server_negotiation.cc:407] Connection header received
0926 16:47:57.744341 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:57.744354 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:57.744781 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:57.745074 (+ 293us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:47:57.746226 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.749046 (+ 2820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.751678 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.753475 (+ 1797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.753695 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:57.753984 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:57.754677 (+ 693us) server_negotiation.cc:299] Negotiation successful
0926 16:47:57.754917 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7889,"thread_start_us":7323,"threads_started":1}
I20260926 16:47:57.755919 26741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:57.734441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:42251 (local address 127.0.0.1:41380)
0926 16:47:57.743941 (+ 9500us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:57.743964 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:57.744106 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:57.745394 (+ 1288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:57.745401 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:57.745443 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:47:57.745947 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:57.745963 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.749198 (+ 3235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.749207 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:57.751387 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:57.751403 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.753868 (+ 2465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.753876 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:57.754199 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:57.754238 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 16:47:57.754353 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 16:47:57.755577 (+ 1224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9324,"thread_start_us":9132,"threads_started":1}
I20260926 16:47:57.760519 26529 tablet_service.cc:1467] Processing CreateTablet for tablet ec47fa7509424950b44b1bbad9682dff (DEFAULT_TABLE table=test-table [id=40a0e827721f4b2891de2dcd8aadcb95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:47:57.762094 26529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec47fa7509424950b44b1bbad9682dff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:47:57.772972 26744 tablet_bootstrap.cc:492] T ec47fa7509424950b44b1bbad9682dff P 41c3865003b24affa65fae2b870fc44e: Bootstrap starting.
I20260926 16:47:57.776533 26743 tablet_bootstrap.cc:492] T ec47fa7509424950b44b1bbad9682dff P 3c705e0685c14f0f8bfba47d70a3bbec: Bootstrap starting.
I20260926 16:47:57.778918 26745 tablet_bootstrap.cc:492] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6: Bootstrap starting.
I20260926 16:47:57.780747 26744 tablet_bootstrap.cc:654] T ec47fa7509424950b44b1bbad9682dff P 41c3865003b24affa65fae2b870fc44e: Neither blocks nor log segments found. Creating new log.
I20260926 16:47:57.780917 26743 tablet_bootstrap.cc:654] T ec47fa7509424950b44b1bbad9682dff P 3c705e0685c14f0f8bfba47d70a3bbec: Neither blocks nor log segments found. Creating new log.
I20260926 16:47:57.782673 26743 log.cc:824] T ec47fa7509424950b44b1bbad9682dff P 3c705e0685c14f0f8bfba47d70a3bbec: Log is configured to *not* fsync() on all Append() calls
I20260926 16:47:57.783790 26745 tablet_bootstrap.cc:654] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6: Neither blocks nor log segments found. Creating new log.
I20260926 16:47:57.784121 26744 log.cc:824] T ec47fa7509424950b44b1bbad9682dff P 41c3865003b24affa65fae2b870fc44e: Log is configured to *not* fsync() on all Append() calls
I20260926 16:47:57.785348 26745 log.cc:824] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6: Log is configured to *not* fsync() on all Append() calls
I20260926 16:47:57.786259 26743 tablet_bootstrap.cc:492] T ec47fa7509424950b44b1bbad9682dff P 3c705e0685c14f0f8bfba47d70a3bbec: No bootstrap required, opened a new log
I20260926 16:47:57.786602 26743 ts_tablet_manager.cc:1397] T ec47fa7509424950b44b1bbad9682dff P 3c705e0685c14f0f8bfba47d70a3bbec: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260926 16:47:57.788523 26744 tablet_bootstrap.cc:492] T ec47fa7509424950b44b1bbad9682dff P 41c3865003b24affa65fae2b870fc44e: No bootstrap required, opened a new log
I20260926 16:47:57.788887 26745 tablet_bootstrap.cc:492] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6: No bootstrap required, opened a new log
I20260926 16:47:57.788945 26744 ts_tablet_manager.cc:1397] T ec47fa7509424950b44b1bbad9682dff P 41c3865003b24affa65fae2b870fc44e: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.000s
I20260926 16:47:57.789197 26745 ts_tablet_manager.cc:1397] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260926 16:47:57.794845 26743 raft_consensus.cc:348] T ec47fa7509424950b44b1bbad9682dff P 3c705e0685c14f0f8bfba47d70a3bbec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c705e0685c14f0f8bfba47d70a3bbec" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44745 } } peers { permanent_uuid: "41c3865003b24affa65fae2b870fc44e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46641 } } peers { permanent_uuid: "e57ee7c944b64744b83764bb53c16bf6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42251 } }
I20260926 16:47:57.795288 26743 raft_consensus.cc:374] T ec47fa7509424950b44b1bbad9682dff P 3c705e0685c14f0f8bfba47d70a3bbec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:47:57.795378 26743 raft_consensus.cc:729] T ec47fa7509424950b44b1bbad9682dff P 3c705e0685c14f0f8bfba47d70a3bbec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c705e0685c14f0f8bfba47d70a3bbec, State: Initialized, Role: FOLLOWER
I20260926 16:47:57.796097 26743 consensus_queue.cc:260] T ec47fa7509424950b44b1bbad9682dff P 3c705e0685c14f0f8bfba47d70a3bbec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c705e0685c14f0f8bfba47d70a3bbec" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44745 } } peers { permanent_uuid: "41c3865003b24affa65fae2b870fc44e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46641 } } peers { permanent_uuid: "e57ee7c944b64744b83764bb53c16bf6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42251 } }
I20260926 16:47:57.797827 26743 ts_tablet_manager.cc:1428] T ec47fa7509424950b44b1bbad9682dff P 3c705e0685c14f0f8bfba47d70a3bbec: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260926 16:47:57.801580 26744 raft_consensus.cc:348] T ec47fa7509424950b44b1bbad9682dff P 41c3865003b24affa65fae2b870fc44e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c705e0685c14f0f8bfba47d70a3bbec" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44745 } } peers { permanent_uuid: "41c3865003b24affa65fae2b870fc44e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46641 } } peers { permanent_uuid: "e57ee7c944b64744b83764bb53c16bf6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42251 } }
I20260926 16:47:57.802228 26744 raft_consensus.cc:374] T ec47fa7509424950b44b1bbad9682dff P 41c3865003b24affa65fae2b870fc44e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:47:57.802392 26744 raft_consensus.cc:729] T ec47fa7509424950b44b1bbad9682dff P 41c3865003b24affa65fae2b870fc44e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c3865003b24affa65fae2b870fc44e, State: Initialized, Role: FOLLOWER
I20260926 16:47:57.802229 26745 raft_consensus.cc:348] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c705e0685c14f0f8bfba47d70a3bbec" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44745 } } peers { permanent_uuid: "41c3865003b24affa65fae2b870fc44e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46641 } } peers { permanent_uuid: "e57ee7c944b64744b83764bb53c16bf6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42251 } }
I20260926 16:47:57.802956 26745 raft_consensus.cc:374] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:47:57.803092 26745 raft_consensus.cc:729] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e57ee7c944b64744b83764bb53c16bf6, State: Initialized, Role: FOLLOWER
I20260926 16:47:57.803833 26745 consensus_queue.cc:260] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c705e0685c14f0f8bfba47d70a3bbec" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44745 } } peers { permanent_uuid: "41c3865003b24affa65fae2b870fc44e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46641 } } peers { permanent_uuid: "e57ee7c944b64744b83764bb53c16bf6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42251 } }
I20260926 16:47:57.803916 26744 consensus_queue.cc:260] T ec47fa7509424950b44b1bbad9682dff P 41c3865003b24affa65fae2b870fc44e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c705e0685c14f0f8bfba47d70a3bbec" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44745 } } peers { permanent_uuid: "41c3865003b24affa65fae2b870fc44e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46641 } } peers { permanent_uuid: "e57ee7c944b64744b83764bb53c16bf6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42251 } }
I20260926 16:47:57.808348 26594 heartbeater.cc:503] Master 127.9.238.254:35375 was elected leader, sending a full tablet report...
I20260926 16:47:57.808485 26726 heartbeater.cc:503] Master 127.9.238.254:35375 was elected leader, sending a full tablet report...
I20260926 16:47:57.809106 26745 ts_tablet_manager.cc:1428] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6: Time spent starting tablet: real 0.020s user 0.015s sys 0.004s
I20260926 16:47:57.809894 26744 ts_tablet_manager.cc:1428] T ec47fa7509424950b44b1bbad9682dff P 41c3865003b24affa65fae2b870fc44e: Time spent starting tablet: real 0.021s user 0.019s sys 0.000s
I20260926 16:47:57.836402 26750 raft_consensus.cc:482] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:47:57.836941 26750 raft_consensus.cc:504] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c705e0685c14f0f8bfba47d70a3bbec" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44745 } } peers { permanent_uuid: "41c3865003b24affa65fae2b870fc44e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46641 } } peers { permanent_uuid: "e57ee7c944b64744b83764bb53c16bf6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42251 } }
I20260926 16:47:57.839859 26750 leader_election.cc:288] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c705e0685c14f0f8bfba47d70a3bbec (127.9.238.193:44745), 41c3865003b24affa65fae2b870fc44e (127.9.238.195:46641)
I20260926 16:47:57.849619 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:57.841350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:50317 (local address 127.9.238.195:46641)
0926 16:47:57.841727 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:57.841736 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:57.842584 (+ 848us) server_negotiation.cc:407] Connection header received
0926 16:47:57.842995 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:57.843001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:57.843094 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:57.843191 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:47:57.844115 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.844861 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.847362 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.848424 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.848729 (+ 305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:57.848818 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:57.849217 (+ 399us) server_negotiation.cc:299] Negotiation successful
0926 16:47:57.849350 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260926 16:47:57.849833 26753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:57.841195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:46641 (local address 127.9.238.194:50317)
0926 16:47:57.842428 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:57.842450 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:57.842742 (+ 292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:57.843285 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:57.843291 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:57.843328 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:47:57.843863 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:57.843882 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.845047 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.845054 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:57.847170 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:57.847189 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.848768 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.848777 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:57.848959 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:57.848988 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 16:47:57.849112 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 16:47:57.849271 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1063,"mutex_wait_us":118,"spinlock_wait_cycles":32000,"thread_start_us":847,"threads_started":1}
I20260926 16:47:57.850613 26737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:57.840278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:47035 (local address 127.9.238.193:44745)
0926 16:47:57.840674 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:57.840686 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:57.841376 (+ 690us) server_negotiation.cc:407] Connection header received
0926 16:47:57.841823 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:57.841831 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:57.841939 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:57.842027 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:47:57.843231 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.844523 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.847432 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.848965 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.849221 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:57.849315 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:57.850176 (+ 861us) server_negotiation.cc:299] Negotiation successful
0926 16:47:57.850341 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260926 16:47:57.850970 26752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:57.840216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:44745 (local address 127.9.238.194:47035)
0926 16:47:57.841130 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0926 16:47:57.841169 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 16:47:57.841565 (+ 396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:47:57.842318 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:47:57.842324 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:47:57.842354 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:47:57.843008 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:57.843030 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.844717 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.844724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:57.847155 (+ 2431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:47:57.847178 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.849327 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.849338 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:47:57.849814 (+ 476us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:57.849883 (+ 69us) client_negotiation.cc:769] Sending connection context
0926 16:47:57.850101 (+ 218us) client_negotiation.cc:240] Negotiation successful
0926 16:47:57.850334 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":8,"thread_start_us":483,"threads_started":1}
I20260926 16:47:57.851393 26681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec47fa7509424950b44b1bbad9682dff" candidate_uuid: "e57ee7c944b64744b83764bb53c16bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c3865003b24affa65fae2b870fc44e" is_pre_election: true
I20260926 16:47:57.852331 26681 raft_consensus.cc:2454] T ec47fa7509424950b44b1bbad9682dff P 41c3865003b24affa65fae2b870fc44e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e57ee7c944b64744b83764bb53c16bf6 in term 0.
I20260926 16:47:57.852246 26417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec47fa7509424950b44b1bbad9682dff" candidate_uuid: "e57ee7c944b64744b83764bb53c16bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c705e0685c14f0f8bfba47d70a3bbec" is_pre_election: true
I20260926 16:47:57.852921 26417 raft_consensus.cc:2454] T ec47fa7509424950b44b1bbad9682dff P 3c705e0685c14f0f8bfba47d70a3bbec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e57ee7c944b64744b83764bb53c16bf6 in term 0.
I20260926 16:47:57.853533 26482 leader_election.cc:302] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41c3865003b24affa65fae2b870fc44e, e57ee7c944b64744b83764bb53c16bf6; no voters:
I20260926 16:47:57.854105 26750 raft_consensus.cc:2781] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:47:57.854339 26750 raft_consensus.cc:482] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:47:57.854460 26750 raft_consensus.cc:3037] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:47:57.856938 26750 raft_consensus.cc:504] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c705e0685c14f0f8bfba47d70a3bbec" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44745 } } peers { permanent_uuid: "41c3865003b24affa65fae2b870fc44e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46641 } } peers { permanent_uuid: "e57ee7c944b64744b83764bb53c16bf6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42251 } }
I20260926 16:47:57.857628 26750 leader_election.cc:288] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6 [CANDIDATE]: Term 1 election: Requested vote from peers 3c705e0685c14f0f8bfba47d70a3bbec (127.9.238.193:44745), 41c3865003b24affa65fae2b870fc44e (127.9.238.195:46641)
I20260926 16:47:57.858443 26417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec47fa7509424950b44b1bbad9682dff" candidate_uuid: "e57ee7c944b64744b83764bb53c16bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c705e0685c14f0f8bfba47d70a3bbec"
I20260926 16:47:57.858721 26417 raft_consensus.cc:3037] T ec47fa7509424950b44b1bbad9682dff P 3c705e0685c14f0f8bfba47d70a3bbec [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:47:57.858672 26681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec47fa7509424950b44b1bbad9682dff" candidate_uuid: "e57ee7c944b64744b83764bb53c16bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c3865003b24affa65fae2b870fc44e"
I20260926 16:47:57.859040 26681 raft_consensus.cc:3037] T ec47fa7509424950b44b1bbad9682dff P 41c3865003b24affa65fae2b870fc44e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:47:57.861361 26417 raft_consensus.cc:2454] T ec47fa7509424950b44b1bbad9682dff P 3c705e0685c14f0f8bfba47d70a3bbec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e57ee7c944b64744b83764bb53c16bf6 in term 1.
I20260926 16:47:57.861922 26681 raft_consensus.cc:2454] T ec47fa7509424950b44b1bbad9682dff P 41c3865003b24affa65fae2b870fc44e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e57ee7c944b64744b83764bb53c16bf6 in term 1.
I20260926 16:47:57.861979 26482 leader_election.cc:302] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c705e0685c14f0f8bfba47d70a3bbec, e57ee7c944b64744b83764bb53c16bf6; no voters:
I20260926 16:47:57.862404 26750 raft_consensus.cc:2781] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:47:57.863008 26750 raft_consensus.cc:686] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6 [term 1 LEADER]: Becoming Leader. State: Replica: e57ee7c944b64744b83764bb53c16bf6, State: Running, Role: LEADER
I20260926 16:47:57.863597 26750 consensus_queue.cc:237] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c705e0685c14f0f8bfba47d70a3bbec" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44745 } } peers { permanent_uuid: "41c3865003b24affa65fae2b870fc44e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46641 } } peers { permanent_uuid: "e57ee7c944b64744b83764bb53c16bf6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42251 } }
I20260926 16:47:57.869439 26276 catalog_manager.cc:5495] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e57ee7c944b64744b83764bb53c16bf6 (127.9.238.194). New cstate: current_term: 1 leader_uuid: "e57ee7c944b64744b83764bb53c16bf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c705e0685c14f0f8bfba47d70a3bbec" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41c3865003b24affa65fae2b870fc44e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e57ee7c944b64744b83764bb53c16bf6" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 42251 } health_report { overall_health: HEALTHY } } }
W20260926 16:47:57.871637 26727 tablet.cc:2369] T ec47fa7509424950b44b1bbad9682dff P 41c3865003b24affa65fae2b870fc44e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:47:57.913084 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:57.906386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41390 (local address 127.9.238.194:42251)
0926 16:47:57.906616 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:57.906621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:57.906641 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:47:57.906756 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:57.906760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:57.906843 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:57.906986 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:47:57.907519 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.908307 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.910525 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.911001 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.911164 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:57.911820 (+ 656us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:47:57.912389 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:47:57.912521 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:57.912652 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 16:47:57.912730 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 16:47:57.923588 26417 raft_consensus.cc:1261] T ec47fa7509424950b44b1bbad9682dff P 3c705e0685c14f0f8bfba47d70a3bbec [term 1 FOLLOWER]: Refusing update from remote peer e57ee7c944b64744b83764bb53c16bf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:47:57.923781 26681 raft_consensus.cc:1261] T ec47fa7509424950b44b1bbad9682dff P 41c3865003b24affa65fae2b870fc44e [term 1 FOLLOWER]: Refusing update from remote peer e57ee7c944b64744b83764bb53c16bf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:47:57.924758 26754 consensus_queue.cc:1035] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c705e0685c14f0f8bfba47d70a3bbec" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:47:57.925206 26750 consensus_queue.cc:1035] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c3865003b24affa65fae2b870fc44e" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:47:57.939109 26759 mvcc.cc:205] Tried to move back new op lower bound from 7333647474362621952 to 7333647474134142976. Current Snapshot: MvccSnapshot[applied={T|T < 7333647474362621952}]
I20260926 16:47:57.939493 26757 mvcc.cc:205] Tried to move back new op lower bound from 7333647474362621952 to 7333647474134142976. Current Snapshot: MvccSnapshot[applied={T|T < 7333647474362621952}]
I20260926 16:47:57.956060 26277 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:47:57.956460 26277 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:47:57.961061 26277 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4: Sending DeleteTablet for 3 replicas of tablet ec47fa7509424950b44b1bbad9682dff
I20260926 16:47:57.962858 26661 tablet_service.cc:1514] Processing DeleteTablet for tablet ec47fa7509424950b44b1bbad9682dff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:47:57 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:57336
I20260926 16:47:57.963100 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 26314
I20260926 16:47:57.963088 26529 tablet_service.cc:1514] Processing DeleteTablet for tablet ec47fa7509424950b44b1bbad9682dff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:47:57 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:41380
I20260926 16:47:57.963088 26397 tablet_service.cc:1514] Processing DeleteTablet for tablet ec47fa7509424950b44b1bbad9682dff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:47:57 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:55334
I20260926 16:47:57.975699 26767 tablet_replica.cc:330] T ec47fa7509424950b44b1bbad9682dff P 41c3865003b24affa65fae2b870fc44e: stopping tablet replica
I20260926 16:47:57.976078 26768 tablet_replica.cc:330] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6: stopping tablet replica
I20260926 16:47:57.976444 26767 raft_consensus.cc:2229] T ec47fa7509424950b44b1bbad9682dff P 41c3865003b24affa65fae2b870fc44e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:47:57.976641 26768 raft_consensus.cc:2229] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:47:57.977195 26767 raft_consensus.cc:2258] T ec47fa7509424950b44b1bbad9682dff P 41c3865003b24affa65fae2b870fc44e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:47:57.977488 26768 raft_consensus.cc:2258] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:47:57.978600 26769 tablet_replica.cc:330] T ec47fa7509424950b44b1bbad9682dff P 3c705e0685c14f0f8bfba47d70a3bbec: stopping tablet replica
I20260926 16:47:57.979152 26769 raft_consensus.cc:2229] T ec47fa7509424950b44b1bbad9682dff P 3c705e0685c14f0f8bfba47d70a3bbec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:47:57.979697 26769 raft_consensus.cc:2258] T ec47fa7509424950b44b1bbad9682dff P 3c705e0685c14f0f8bfba47d70a3bbec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:47:57.980734 26767 ts_tablet_manager.cc:1905] T ec47fa7509424950b44b1bbad9682dff P 41c3865003b24affa65fae2b870fc44e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:47:57.982270 26769 ts_tablet_manager.cc:1905] T ec47fa7509424950b44b1bbad9682dff P 3c705e0685c14f0f8bfba47d70a3bbec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:47:57.988539 26768 ts_tablet_manager.cc:1905] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:47:57.989725 26767 ts_tablet_manager.cc:1918] T ec47fa7509424950b44b1bbad9682dff P 41c3865003b24affa65fae2b870fc44e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:47:57.990044 26767 log.cc:1196] T ec47fa7509424950b44b1bbad9682dff P 41c3865003b24affa65fae2b870fc44e: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/wal/wals/ec47fa7509424950b44b1bbad9682dff
I20260926 16:47:57.990444 26769 ts_tablet_manager.cc:1918] T ec47fa7509424950b44b1bbad9682dff P 3c705e0685c14f0f8bfba47d70a3bbec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:47:57.990675 26769 log.cc:1196] T ec47fa7509424950b44b1bbad9682dff P 3c705e0685c14f0f8bfba47d70a3bbec: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/wal/wals/ec47fa7509424950b44b1bbad9682dff
I20260926 16:47:57.990886 26767 ts_tablet_manager.cc:1939] T ec47fa7509424950b44b1bbad9682dff P 41c3865003b24affa65fae2b870fc44e: Deleting consensus metadata
I20260926 16:47:57.991390 26769 ts_tablet_manager.cc:1939] T ec47fa7509424950b44b1bbad9682dff P 3c705e0685c14f0f8bfba47d70a3bbec: Deleting consensus metadata
I20260926 16:47:57.993525 26261 catalog_manager.cc:4841] TS 41c3865003b24affa65fae2b870fc44e (127.9.238.195:46641): tablet ec47fa7509424950b44b1bbad9682dff (table test-table [id=40a0e827721f4b2891de2dcd8aadcb95]) successfully deleted
I20260926 16:47:57.993994 26261 catalog_manager.cc:4841] TS 3c705e0685c14f0f8bfba47d70a3bbec (127.9.238.193:44745): tablet ec47fa7509424950b44b1bbad9682dff (table test-table [id=40a0e827721f4b2891de2dcd8aadcb95]) successfully deleted
I20260926 16:47:57.996001 26737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:57.965635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55348 (local address 127.9.238.193:44745)
0926 16:47:57.965913 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:57.965923 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:57.965953 (+ 30us) server_negotiation.cc:407] Connection header received
0926 16:47:57.966054 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:57.966065 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:57.966193 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:57.966364 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:57.969926 (+ 3562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.971664 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.975442 (+ 3778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:57.976152 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:57.981298 (+ 5146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:57.981649 (+ 351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:57.981664 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:57.981687 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:57.981753 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:57.986377 (+ 4624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:57.993012 (+ 6635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:57.993023 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:57.993026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:57.993303 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:57.993865 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:57.993870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:57.993873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:57.994417 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:57.994781 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:57.995111 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 16:47:57.995407 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260926 16:47:57.996807 26457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:47:58.000438 26768 ts_tablet_manager.cc:1918] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:47:58.000707 26768 log.cc:1196] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/wal/wals/ec47fa7509424950b44b1bbad9682dff
I20260926 16:47:58.001511 26768 ts_tablet_manager.cc:1939] T ec47fa7509424950b44b1bbad9682dff P e57ee7c944b64744b83764bb53c16bf6: Deleting consensus metadata
I20260926 16:47:58.004195 26261 catalog_manager.cc:4841] TS e57ee7c944b64744b83764bb53c16bf6 (127.9.238.194:42251): tablet ec47fa7509424950b44b1bbad9682dff (table test-table [id=40a0e827721f4b2891de2dcd8aadcb95]) successfully deleted
I20260926 16:47:58.109339 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 26314
I20260926 16:47:58.132004 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 26466
I20260926 16:47:58.152699 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:58.133096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41396 (local address 127.9.238.194:42251)
0926 16:47:58.133362 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:58.133369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:58.133405 (+ 36us) server_negotiation.cc:407] Connection header received
0926 16:47:58.133685 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:58.133693 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:58.133793 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:58.133962 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:58.136444 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:58.137614 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:58.139117 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:58.139947 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:58.140110 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:58.143487 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:58.143508 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:58.143535 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:58.143647 (+ 112us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:58.148578 (+ 4931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:58.149652 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:58.149664 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:58.149666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:58.149925 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:58.150415 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:58.150422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:58.150426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:58.151012 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:58.151487 (+ 475us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:58.151811 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 16:47:58.152125 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260926 16:47:58.153676 26589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:47:58.289371 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 26466
I20260926 16:47:58.314327 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 26598
I20260926 16:47:58.332150 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:47:58.315889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57344 (local address 127.9.238.195:46641)
0926 16:47:58.316475 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0926 16:47:58.316480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:47:58.316505 (+ 25us) server_negotiation.cc:407] Connection header received
0926 16:47:58.316572 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:47:58.316578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:47:58.316653 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:47:58.316773 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:47:58.318444 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:58.319282 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:58.321040 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:47:58.321477 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:47:58.321681 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:47:58.325097 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:47:58.325115 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:47:58.325133 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:47:58.325181 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:47:58.328620 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:58.329621 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:58.329629 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:58.329632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:58.329789 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:47:58.330392 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:47:58.330396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:47:58.330398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:47:58.330804 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:47:58.331050 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0926 16:47:58.331483 (+ 433us) server_negotiation.cc:299] Negotiation successful
0926 16:47:58.331780 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260926 16:47:58.332791 26721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:47:58.438205 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 26598
I20260926 16:47:58.456646 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 26244
I20260926 16:47:58.457902 26307 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:47:58.596426 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 26244
I20260926 16:48:18.624372 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:35375
--webserver_interface=127.9.238.254
--webserver_port=39433
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:45093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:35375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:48:19.048372 26774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:48:19.048804 26774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:48:19.049006 26774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:48:19.057644 26774 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 16:48:19.057819 26774 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 16:48:19.057917 26774 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 16:48:19.058024 26774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:48:19.058136 26774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:48:19.058274 26774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:48:19.058364 26774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:48:19.058452 26774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:48:19.071434 26774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:45093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:35375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:35375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=39433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:48:19.073118 26774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:48:19.074697 26774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:48:19.089224 26782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:19.089314 26780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:19.089457 26779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:48:19.089731 26774 server_base.cc:1034] running on GCE node
I20260926 16:48:19.091166 26774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:48:19.093236 26774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:48:19.094586 26774 hybrid_clock.cc:648] HybridClock initialized: now 1790441299094480 us; error 93 us; skew 500 ppm
I20260926 16:48:19.095130 26774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:48:19 dist-test-slave-wt2j krb5kdc[25690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441299, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:48:19.099677 26774 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:48:19.101718 26774 webserver.cc:460] Webserver started at http://127.9.238.254:39433/ using document root <none> and password file <none>
I20260926 16:48:19.102674 26774 fs_manager.cc:359] Metadata directory not provided
I20260926 16:48:19.102842 26774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:48:19.110530 26774 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:48:19.114857 26789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:19.116986 26774 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:48:19.117264 26774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "367b3ecfd4f3444999c5e41fa619c7c4"
format_stamp: "Formatted at 2026-09-26 16:47:32 on dist-test-slave-wt2j"
server_key: "9e3f04b9ec02587a3063bbf571fe0483"
server_key_iv: "14edfba153f0ac42caa1f6136a40b942"
server_key_version: "encryptionkey@0"
I20260926 16:48:19.118621 26774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:48:19.145068 26774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:48:19.150473 26774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:48:19.150861 26774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:48:19.176036 26774 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:35375
I20260926 16:48:19.176115 26840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:35375 every 8 connection(s)
I20260926 16:48:19.178228 26774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:48:19.185959 26841 sys_catalog.cc:263] Verifying existing consensus state
I20260926 16:48:19.187521 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 26774
I20260926 16:48:19.188690 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:44745
--local_ip_for_outbound_sockets=127.9.238.193
--tserver_master_addrs=127.9.238.254:35375
--webserver_port=37245
--webserver_interface=127.9.238.193
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:45093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:48:19.189669 26841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4: Bootstrap starting.
I20260926 16:48:19.210876 26841 log.cc:824] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4: Log is configured to *not* fsync() on all Append() calls
I20260926 16:48:19.227862 26841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 16:48:19.228497 26841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4: Bootstrap complete.
I20260926 16:48:19.238245 26841 raft_consensus.cc:348] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367b3ecfd4f3444999c5e41fa619c7c4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35375 } }
I20260926 16:48:19.239284 26841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 367b3ecfd4f3444999c5e41fa619c7c4, State: Initialized, Role: FOLLOWER
I20260926 16:48:19.240091 26841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367b3ecfd4f3444999c5e41fa619c7c4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35375 } }
I20260926 16:48:19.240350 26841 raft_consensus.cc:388] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:48:19.240545 26841 raft_consensus.cc:482] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:48:19.240782 26841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [term 2 FOLLOWER]: Advancing to term 3
I20260926 16:48:19.245205 26841 raft_consensus.cc:504] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367b3ecfd4f3444999c5e41fa619c7c4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35375 } }
I20260926 16:48:19.245765 26841 leader_election.cc:302] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 367b3ecfd4f3444999c5e41fa619c7c4; no voters:
I20260926 16:48:19.246495 26841 leader_election.cc:288] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 16:48:19.246829 26845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 16:48:19.248219 26845 raft_consensus.cc:686] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [term 3 LEADER]: Becoming Leader. State: Replica: 367b3ecfd4f3444999c5e41fa619c7c4, State: Running, Role: LEADER
I20260926 16:48:19.249004 26841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:48:19.249172 26845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367b3ecfd4f3444999c5e41fa619c7c4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35375 } }
I20260926 16:48:19.252564 26847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 367b3ecfd4f3444999c5e41fa619c7c4. Latest consensus state: current_term: 3 leader_uuid: "367b3ecfd4f3444999c5e41fa619c7c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367b3ecfd4f3444999c5e41fa619c7c4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35375 } } }
I20260926 16:48:19.253147 26847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:48:19.253880 26846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "367b3ecfd4f3444999c5e41fa619c7c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "367b3ecfd4f3444999c5e41fa619c7c4" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35375 } } }
I20260926 16:48:19.254297 26846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:48:19.265082 26851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:48:19.275389 26851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:48:19.277216 26851 catalog_manager.cc:1259] Loaded cluster ID: 6fc673e8d4434f8997712dcaed43249a
I20260926 16:48:19.277534 26851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:48:19.283491 26851 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:48:19.285715 26851 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4: Loaded TSK: 0
I20260926 16:48:19.286042 26851 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 367b3ecfd4f3444999c5e41fa619c7c4: Loaded TSK: 1
F20260926 16:48:19.287412 26851 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdb247769ad google::LogMessage::Fail() at ??:0
@ 0x7fdb2477988a google::LogMessage::SendToLog() at ??:0
@ 0x7fdb247764de google::LogMessage::Flush() at ??:0
@ 0x7fdb24777a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdb284da7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fdb284c973e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fdb3a1a51c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fdb3a1a4783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fdb3a2302a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fdb3a2300c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fdb3a27a5e9 std::function<>::operator()() at ??:0
@ 0x7fdb3a1a8cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fdb3a1a6c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fdb3a22e610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fdb3a22e439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fdb273c2345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdb273c591c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fdb273c5799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fdb273a0eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdb2a67a6db start_thread at ??:0
@ 0x7fdb2361061f clone at ??:0
W20260926 16:48:19.693261 26843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:48:19.693653 26843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:48:19.693835 26843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:48:19.702318 26843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:48:19.702497 26843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:48:19.702725 26843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:48:19.714385 26843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:45093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:44745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=37245
--tserver_master_addrs=127.9.238.254:35375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:48:19.716326 26843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:48:19.718160 26843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:48:19.735643 26888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:19.735594 26891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:19.735791 26889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:48:19.736326 26843 server_base.cc:1034] running on GCE node
I20260926 16:48:19.737437 26843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:48:19.738772 26843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:48:19.740056 26843 hybrid_clock.cc:648] HybridClock initialized: now 1790441299739970 us; error 84 us; skew 500 ppm
I20260926 16:48:19.740567 26843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:48:19 dist-test-slave-wt2j krb5kdc[25690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441299, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:48:19.745052 26843 init.cc:377] Logged in from keytab as kudu/127.9.238.193@KRBTEST.COM (short username kudu)
I20260926 16:48:19.747354 26843 webserver.cc:460] Webserver started at http://127.9.238.193:37245/ using document root <none> and password file <none>
I20260926 16:48:19.748452 26843 fs_manager.cc:359] Metadata directory not provided
I20260926 16:48:19.748643 26843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:48:19.756534 26843 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260926 16:48:19.761569 26898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:19.763839 26843 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:48:19.764168 26843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "3c705e0685c14f0f8bfba47d70a3bbec"
format_stamp: "Formatted at 2026-09-26 16:47:33 on dist-test-slave-wt2j"
server_key: "21c6914b9ceee487480bd7fac2b6459c"
server_key_iv: "b0369cb9e787e2237e5f2918cd5a66d3"
server_key_version: "encryptionkey@0"
I20260926 16:48:19.765583 26843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:48:19.790369 26843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:48:19.796161 26843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:48:19.796541 26843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:48:19.797910 26843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:48:19.799856 26843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:48:19.800010 26843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:19.800172 26843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:48:19.800278 26843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:19.844918 26843 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:44745
I20260926 16:48:19.845058 27010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:44745 every 8 connection(s)
I20260926 16:48:19.846946 26843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:48:19.853677 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 26843
I20260926 16:48:19.854259 27013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:19.850974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.193:55877)
0926 16:48:19.852290 (+ 1316us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:19.852644 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":859,"thread_start_us":570,"threads_started":1}
I20260926 16:48:19.854682 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:42251
--local_ip_for_outbound_sockets=127.9.238.194
--tserver_master_addrs=127.9.238.254:35375
--webserver_port=41099
--webserver_interface=127.9.238.194
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:45093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:48:19.855240 27011 heartbeater.cc:644] Failed to heartbeat to 127.9.238.254:35375 (0 consecutive failures): Network error: Failed to ping master at 127.9.238.254:35375: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
I20260926 16:48:19.856664 27013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:19.856070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.193:49795)
0926 16:48:19.856339 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:19.856473 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 16:48:19.858013 27013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:19.857471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.193:45767)
0926 16:48:19.857704 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:19.857834 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260926 16:48:19.858366 27011 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 16:48:20.258685 27014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:48:20.259038 27014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:48:20.259198 27014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:48:20.266706 27014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:48:20.266837 27014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:48:20.267040 27014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:48:20.277382 27014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:45093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:42251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=41099
--tserver_master_addrs=127.9.238.254:35375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:48:20.278823 27014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:48:20.280344 27014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:48:20.295656 27019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:20.295723 27020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:20.295642 27022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:48:20.296941 27014 server_base.cc:1034] running on GCE node
I20260926 16:48:20.297657 27014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:48:20.298763 27014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:48:20.300092 27014 hybrid_clock.cc:648] HybridClock initialized: now 1790441300299964 us; error 112 us; skew 500 ppm
I20260926 16:48:20.300473 27014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:48:20 dist-test-slave-wt2j krb5kdc[25690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441300, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:48:20.304315 27014 init.cc:377] Logged in from keytab as kudu/127.9.238.194@KRBTEST.COM (short username kudu)
I20260926 16:48:20.306133 27014 webserver.cc:460] Webserver started at http://127.9.238.194:41099/ using document root <none> and password file <none>
I20260926 16:48:20.306908 27014 fs_manager.cc:359] Metadata directory not provided
I20260926 16:48:20.307063 27014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:48:20.313297 27014 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 16:48:20.316746 27029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:20.318554 27014 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:48:20.318764 27014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "e57ee7c944b64744b83764bb53c16bf6"
format_stamp: "Formatted at 2026-09-26 16:47:33 on dist-test-slave-wt2j"
server_key: "f33fafa982a8b00af00517c6f71a9f84"
server_key_iv: "a6237a5d3f7b619cb2b1e178675251b4"
server_key_version: "encryptionkey@0"
I20260926 16:48:20.319952 27014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:48:20.343777 27014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:48:20.348519 27014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:48:20.348821 27014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:48:20.349967 27014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:48:20.351701 27014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:48:20.351843 27014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:20.351975 27014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:48:20.352093 27014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:20.392309 27014 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:42251
I20260926 16:48:20.392372 27141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:42251 every 8 connection(s)
I20260926 16:48:20.393827 27014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:48:20.399369 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 27014
I20260926 16:48:20.400249 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:46641
--local_ip_for_outbound_sockets=127.9.238.195
--tserver_master_addrs=127.9.238.254:35375
--webserver_port=41093
--webserver_interface=127.9.238.195
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:45093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:48:20.400347 27144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:20.397745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.194:43595)
0926 16:48:20.398769 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:20.399099 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":389,"threads_started":1}
W20260926 16:48:20.401357 27142 heartbeater.cc:644] Failed to heartbeat to 127.9.238.254:35375 (0 consecutive failures): Network error: Failed to ping master at 127.9.238.254:35375: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
I20260926 16:48:20.402642 27144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:20.402186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.194:38227)
0926 16:48:20.402424 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:20.402518 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260926 16:48:20.403878 27144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:20.403384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.194:48689)
0926 16:48:20.403636 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:20.403739 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260926 16:48:20.404237 27142 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 16:48:20.754180 27145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:48:20.754487 27145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:48:20.754639 27145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:48:20.762130 27145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:48:20.762251 27145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:48:20.762442 27145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:48:20.772603 27145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:45093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:46641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=41093
--tserver_master_addrs=127.9.238.254:35375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:48:20.773962 27145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:48:20.775305 27145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:48:20.790242 27151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:20.790279 27153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:20.790371 27150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:48:20.792166 27145 server_base.cc:1034] running on GCE node
I20260926 16:48:20.792935 27145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:48:20.794117 27145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:48:20.795486 27145 hybrid_clock.cc:648] HybridClock initialized: now 1790441300795391 us; error 93 us; skew 500 ppm
I20260926 16:48:20.795877 27145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:48:20 dist-test-slave-wt2j krb5kdc[25690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441300, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:48:20.807682 27145 init.cc:377] Logged in from keytab as kudu/127.9.238.195@KRBTEST.COM (short username kudu)
I20260926 16:48:20.809615 27145 webserver.cc:460] Webserver started at http://127.9.238.195:41093/ using document root <none> and password file <none>
I20260926 16:48:20.810370 27145 fs_manager.cc:359] Metadata directory not provided
I20260926 16:48:20.810534 27145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:48:20.816811 27145 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 16:48:20.820257 27160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:20.821883 27145 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 16:48:20.822109 27145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "41c3865003b24affa65fae2b870fc44e"
format_stamp: "Formatted at 2026-09-26 16:47:34 on dist-test-slave-wt2j"
server_key: "b955bf0b553f3596a6d5c8b88ad373f7"
server_key_iv: "8791de76ad05e607fcf0af47da8ab019"
server_key_version: "encryptionkey@0"
I20260926 16:48:20.823150 27145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:48:20.836730 27145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:48:20.841914 27145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:48:20.842259 27145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:48:20.843441 27145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:48:20.845250 27145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:48:20.845381 27145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:20.845527 27145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:48:20.845634 27145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:20.860873 27199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:20.859223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.193:57883)
0926 16:48:20.860275 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:20.860462 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":842,"thread_start_us":634,"threads_started":1}
I20260926 16:48:20.885223 27145 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:46641
I20260926 16:48:20.885337 27273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:46641 every 8 connection(s)
I20260926 16:48:20.886803 27145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:48:20.893121 27276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:20.890565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.195:49335)
0926 16:48:20.891516 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:20.891923 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":343,"threads_started":1}
I20260926 16:48:20.893865 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 27145
W20260926 16:48:20.894142 27274 heartbeater.cc:644] Failed to heartbeat to 127.9.238.254:35375 (0 consecutive failures): Network error: Failed to ping master at 127.9.238.254:35375: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
I20260926 16:48:20.895427 27276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:20.894803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.195:54139)
0926 16:48:20.895052 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:20.895178 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 16:48:20.896920 27276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:20.896368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.195:55833)
0926 16:48:20.896656 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:20.896755 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
W20260926 16:48:20.897325 27274 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 16:48:21.406229 27278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:21.405195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.194:39015)
0926 16:48:21.405843 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:21.405963 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":317,"threads_started":1}
I20260926 16:48:21.863152 27279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:21.862192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.193:36151)
0926 16:48:21.862833 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:21.862943 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":283,"threads_started":1}
I20260926 16:48:21.899245 27280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:21.898090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.195:35369)
0926 16:48:21.898860 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:21.898985 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":391,"threads_started":1}
I20260926 16:48:22.408627 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:22.407696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.194:54541)
0926 16:48:22.408320 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:22.408435 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":301,"threads_started":1}
I20260926 16:48:22.865406 27282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:22.864478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.193:51127)
0926 16:48:22.865089 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:22.865196 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":346,"threads_started":1}
I20260926 16:48:22.901453 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:22.900468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.195:41109)
0926 16:48:22.901125 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:22.901268 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":306,"threads_started":1}
I20260926 16:48:23.410828 27284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:23.409842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.194:34587)
0926 16:48:23.410462 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:23.410610 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":360,"threads_started":1}
I20260926 16:48:23.868090 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:23.866875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.193:56759)
0926 16:48:23.867653 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:23.867776 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":452,"threads_started":1}
I20260926 16:48:23.903539 27286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:23.902694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.195:55993)
0926 16:48:23.903265 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:23.903354 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":327,"threads_started":1}
I20260926 16:48:24.413470 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:24.412360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.194:46519)
0926 16:48:24.413070 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:24.413214 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":343,"threads_started":1}
I20260926 16:48:24.870671 27288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:24.869452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.193:57805)
0926 16:48:24.870203 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:24.870404 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":428,"threads_started":1}
I20260926 16:48:24.905871 27289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:24.904857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.195:44183)
0926 16:48:24.905514 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:24.905617 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":339,"threads_started":1}
I20260926 16:48:25.415740 27290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:25.414747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.194:41695)
0926 16:48:25.415302 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:25.415441 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":286,"threads_started":1}
I20260926 16:48:25.873148 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:25.872131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.193:39279)
0926 16:48:25.872788 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:25.872936 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":341,"threads_started":1}
I20260926 16:48:25.908535 27292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:25.907288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.195:54789)
0926 16:48:25.907976 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:25.908096 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":365,"threads_started":1}
I20260926 16:48:26.417878 27293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:26.416990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.194:60269)
0926 16:48:26.417566 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:26.417685 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":281,"threads_started":1}
I20260926 16:48:26.875703 27294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:26.874517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.193:60257)
0926 16:48:26.875265 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:26.875383 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":346,"threads_started":1}
I20260926 16:48:26.910760 27295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:26.909816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.195:57635)
0926 16:48:26.910402 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:26.910537 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":300,"threads_started":1}
I20260926 16:48:27.420234 27296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:27.419049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.194:56443)
0926 16:48:27.419869 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:27.419981 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":282,"threads_started":1}
I20260926 16:48:27.878391 27297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:27.877205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.193:36561)
0926 16:48:27.878045 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:27.878162 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":424,"threads_started":1}
I20260926 16:48:27.913246 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:27.912217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.195:48117)
0926 16:48:27.912844 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:27.912956 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":303,"threads_started":1}
I20260926 16:48:28.422526 27299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:28.421498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.194:55875)
0926 16:48:28.422110 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:28.422243 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":297,"threads_started":1}
I20260926 16:48:28.880714 27300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:28.879676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.193:44439)
0926 16:48:28.880387 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:28.880498 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":382,"threads_started":1}
I20260926 16:48:28.915489 27301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:28.914593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.195:43055)
0926 16:48:28.915222 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:28.915314 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":326,"threads_started":1}
I20260926 16:48:29.424901 27302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:29.423813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.194:50013)
0926 16:48:29.424578 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:29.424679 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":437,"threads_started":1}
I20260926 16:48:29.883245 27303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:29.882140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.193:40413)
0926 16:48:29.882864 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:29.883041 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":388,"threads_started":1}
I20260926 16:48:29.917793 27304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:29.916824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.195:49307)
0926 16:48:29.917428 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:29.917553 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":292,"threads_started":1}
I20260926 16:48:30.427169 27305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:30.426140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.194:54015)
0926 16:48:30.426862 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:30.426975 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":370,"threads_started":1}
I20260926 16:48:30.885506 27306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:30.884607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.193:58531)
0926 16:48:30.885146 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:30.885278 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":246,"threads_started":1}
I20260926 16:48:30.920182 27307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:30.919153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.195:47073)
0926 16:48:30.919839 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:30.919945 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":332,"threads_started":1}
I20260926 16:48:31.429561 27308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:31.428544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.194:53367)
0926 16:48:31.429203 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:31.429364 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":302,"threads_started":1}
I20260926 16:48:31.887986 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:31.886832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.193:38133)
0926 16:48:31.887664 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:31.887772 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":453,"threads_started":1}
I20260926 16:48:31.922539 27310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:31.921545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.195:55041)
0926 16:48:31.922229 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:31.922347 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":374,"threads_started":1}
I20260926 16:48:32.431752 27311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:32.430748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.194:54863)
0926 16:48:32.431339 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:32.431468 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":281,"threads_started":1}
I20260926 16:48:32.890172 27312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:32.889155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.193:53533)
0926 16:48:32.889773 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:32.889923 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":314,"threads_started":1}
I20260926 16:48:32.925145 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:32.923976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.195:39289)
0926 16:48:32.924768 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:32.924910 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":389,"threads_started":1}
I20260926 16:48:33.434048 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:33.432986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.194:60007)
0926 16:48:33.433633 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:33.433772 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":297,"threads_started":1}
I20260926 16:48:33.893198 27315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:33.891993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.193:57405)
0926 16:48:33.892741 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:33.892910 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":376,"threads_started":1}
I20260926 16:48:33.927599 27316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:33.926453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.195:45165)
0926 16:48:33.927167 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:33.927303 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":357,"threads_started":1}
I20260926 16:48:34.436409 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:34.435134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.194:60379)
0926 16:48:34.435942 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:34.436067 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":439,"threads_started":1}
I20260926 16:48:34.896157 27318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:34.894949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.193:52667)
0926 16:48:34.895783 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:34.895941 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":380,"threads_started":1}
I20260926 16:48:34.929822 27319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:34.928759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.195:41717)
0926 16:48:34.929449 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:34.929573 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":315,"threads_started":1}
I20260926 16:48:35.439103 27320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:35.438067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.194:46801)
0926 16:48:35.438690 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:35.438844 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":326,"threads_started":1}
I20260926 16:48:35.898437 27321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:35.897536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.193:55311)
0926 16:48:35.898135 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:35.898262 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":289,"threads_started":1}
I20260926 16:48:35.907358 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 26843
I20260926 16:48:35.932376 27323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:35.931296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:35375 (local address 127.9.238.195:59899)
0926 16:48:35.932011 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:35.932175 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.254:35375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":422,"threads_started":1}
I20260926 16:48:36.032367 27322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:35.909070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47486 (local address 127.9.238.193:44745)
0926 16:48:35.909912 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:35.909930 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:36.012693 (+102763us) server_negotiation.cc:407] Connection header received
0926 16:48:36.014988 (+ 2295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:36.014999 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:36.015540 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:36.015957 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:36.017720 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:36.018999 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:36.020592 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:36.021294 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:36.021451 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:36.024848 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:36.024857 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:36.024875 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:48:36.024929 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:36.028396 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:36.029382 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:36.029390 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:36.029397 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:36.029661 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:36.030239 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:36.030243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:36.030246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:36.030834 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:36.031111 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:36.031673 (+ 562us) server_negotiation.cc:299] Negotiation successful
0926 16:48:36.032053 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":456,"threads_started":1}
I20260926 16:48:36.034034 27006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:48:36.132176 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 26843
I20260926 16:48:36.152747 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 27014
I20260926 16:48:36.176285 27325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:36.154111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44642 (local address 127.9.238.194:42251)
0926 16:48:36.154882 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:36.154903 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:36.154956 (+ 53us) server_negotiation.cc:407] Connection header received
0926 16:48:36.157119 (+ 2163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:36.157134 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:36.157779 (+ 645us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:36.158206 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:36.160085 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:36.161880 (+ 1795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:36.163416 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:36.164454 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:36.164629 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:36.167824 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:36.167834 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:36.167866 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:48:36.167933 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:36.171933 (+ 4000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:36.172906 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:36.172916 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:36.172926 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:36.173149 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:36.173575 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:36.173578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:36.173581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:36.174360 (+ 779us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:36.174656 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:36.175258 (+ 602us) server_negotiation.cc:299] Negotiation successful
0926 16:48:36.175836 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":398,"threads_started":1}
I20260926 16:48:36.178188 27137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:48:36.269245 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 27014
I20260926 16:48:36.287721 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 27145
I20260926 16:48:36.307652 27327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:36.289110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48380 (local address 127.9.238.195:46641)
0926 16:48:36.289972 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:36.289988 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:36.290034 (+ 46us) server_negotiation.cc:407] Connection header received
0926 16:48:36.291894 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:36.291904 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:36.292343 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:36.292709 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:36.294228 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:36.295426 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:36.296775 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:36.297473 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:36.297698 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:36.300956 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:36.300965 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:36.300985 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:48:36.301031 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:36.304212 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:36.304947 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:36.304954 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:36.304961 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:36.305123 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:36.305506 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:36.305512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:36.305515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:36.306067 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:36.306393 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:36.306865 (+ 472us) server_negotiation.cc:299] Negotiation successful
0926 16:48:36.307280 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":457,"threads_started":1}
I20260926 16:48:36.309621 27269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:48:36.409124 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 27145
2026-09-26T16:48:36Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65918 ms)
[----------] 30 tests from SecurityITest (327799 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:48:36 dist-test-slave-wt2j krb5kdc[27331](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:48:36 dist-test-slave-wt2j krb5kdc[27331](info): set up 2 sockets
Sep 26 16:48:36 dist-test-slave-wt2j krb5kdc[27331](info): commencing operation
krb5kdc: starting...
W20260926 16:48:37.605937 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.007s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:48:37 dist-test-slave-wt2j krb5kdc[27331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:48:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:48:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:48:37.912864 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:37307
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:37307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:48:38.330948 27344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:48:38.331338 27344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:48:38.331585 27344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:48:38.339742 27344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:48:38.339891 27344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:48:38.339993 27344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:48:38.340093 27344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:48:38.340188 27344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:48:38.351531 27344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:37307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:37307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:48:38.353389 27344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:48:38.355015 27344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:48:38.370149 27349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:38.370222 27350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:38.370188 27352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:48:38.371968 27344 server_base.cc:1034] running on GCE node
I20260926 16:48:38.372923 27344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:48:38.374594 27344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:48:38.375855 27344 hybrid_clock.cc:648] HybridClock initialized: now 1790441318375782 us; error 60 us; skew 500 ppm
I20260926 16:48:38.376240 27344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:48:38 dist-test-slave-wt2j krb5kdc[27331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441318, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:48:38.386754 27344 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:48:38.388861 27344 webserver.cc:460] Webserver started at http://127.9.238.254:45799/ using document root <none> and password file <none>
I20260926 16:48:38.389799 27344 fs_manager.cc:359] Metadata directory not provided
I20260926 16:48:38.389974 27344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:48:38.390342 27344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:48:38.394066 27344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "61a7d244dad1440f9ec932841348b616"
format_stamp: "Formatted at 2026-09-26 16:48:38 on dist-test-slave-wt2j"
server_key: "5b5bfe9b1683376a998277fde4f847e2"
server_key_iv: "0584a5e68f89b8e73581c33cc05bd120"
server_key_version: "encryptionkey@0"
I20260926 16:48:38.394984 27344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "61a7d244dad1440f9ec932841348b616"
format_stamp: "Formatted at 2026-09-26 16:48:38 on dist-test-slave-wt2j"
server_key: "5b5bfe9b1683376a998277fde4f847e2"
server_key_iv: "0584a5e68f89b8e73581c33cc05bd120"
server_key_version: "encryptionkey@0"
I20260926 16:48:38.401764 27344 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 16:48:38.406445 27359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:38.408762 27344 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 16:48:38.409044 27344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "61a7d244dad1440f9ec932841348b616"
format_stamp: "Formatted at 2026-09-26 16:48:38 on dist-test-slave-wt2j"
server_key: "5b5bfe9b1683376a998277fde4f847e2"
server_key_iv: "0584a5e68f89b8e73581c33cc05bd120"
server_key_version: "encryptionkey@0"
I20260926 16:48:38.409296 27344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:48:38.430253 27344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:48:38.434993 27344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:48:38.435297 27344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:48:38.458132 27344 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:37307
I20260926 16:48:38.458243 27410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:37307 every 8 connection(s)
I20260926 16:48:38.459860 27344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:48:38.464417 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 27344
I20260926 16:48:38.464700 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:48:38.465413 27411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:48:38.465622 10171 external_mini_cluster.cc:1428] Setting key 7171d4b13ca91d40b3a85dd7ced26dc8
Sep 26 16:48:38 dist-test-slave-wt2j krb5kdc[27331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:48:38.484973 27411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61a7d244dad1440f9ec932841348b616: Bootstrap starting.
I20260926 16:48:38.489750 27411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61a7d244dad1440f9ec932841348b616: Neither blocks nor log segments found. Creating new log.
I20260926 16:48:38.491289 27411 log.cc:824] T 00000000000000000000000000000000 P 61a7d244dad1440f9ec932841348b616: Log is configured to *not* fsync() on all Append() calls
I20260926 16:48:38.496356 27414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:38.468364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40604 (local address 127.9.238.254:37307)
0926 16:48:38.469436 (+ 1072us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:38.469449 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:38.469505 (+ 56us) server_negotiation.cc:407] Connection header received
0926 16:48:38.471450 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:38.471482 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:38.472250 (+ 768us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:38.472745 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:38.475235 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:38.476850 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:38.478968 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:38.479968 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:38.480209 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:38.486481 (+ 6272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:38.486492 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:38.486512 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:48:38.486568 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:38.490224 (+ 3656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:38.491119 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:38.491128 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:38.491136 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:38.491349 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:38.491827 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:38.491833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:38.491836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:38.492552 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:38.492919 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:38.493660 (+ 741us) server_negotiation.cc:299] Negotiation successful
0926 16:48:38.494368 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":495,"threads_started":1}
I20260926 16:48:38.496404 27411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61a7d244dad1440f9ec932841348b616: No bootstrap required, opened a new log
I20260926 16:48:38.506436 27411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61a7d244dad1440f9ec932841348b616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a7d244dad1440f9ec932841348b616" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 37307 } }
I20260926 16:48:38.506955 27411 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61a7d244dad1440f9ec932841348b616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:48:38.507088 27411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61a7d244dad1440f9ec932841348b616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61a7d244dad1440f9ec932841348b616, State: Initialized, Role: FOLLOWER
I20260926 16:48:38.507855 27411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61a7d244dad1440f9ec932841348b616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a7d244dad1440f9ec932841348b616" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 37307 } }
I20260926 16:48:38.508128 27411 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61a7d244dad1440f9ec932841348b616 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:48:38.508291 27411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61a7d244dad1440f9ec932841348b616 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:48:38.508486 27411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61a7d244dad1440f9ec932841348b616 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:48:38.511348 27411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61a7d244dad1440f9ec932841348b616 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a7d244dad1440f9ec932841348b616" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 37307 } }
I20260926 16:48:38.512136 27411 leader_election.cc:302] T 00000000000000000000000000000000 P 61a7d244dad1440f9ec932841348b616 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61a7d244dad1440f9ec932841348b616; no voters:
I20260926 16:48:38.513034 27411 leader_election.cc:288] T 00000000000000000000000000000000 P 61a7d244dad1440f9ec932841348b616 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:48:38.513253 27416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61a7d244dad1440f9ec932841348b616 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:48:38.514920 27416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61a7d244dad1440f9ec932841348b616 [term 1 LEADER]: Becoming Leader. State: Replica: 61a7d244dad1440f9ec932841348b616, State: Running, Role: LEADER
I20260926 16:48:38.516008 27411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61a7d244dad1440f9ec932841348b616 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:48:38.515816 27416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61a7d244dad1440f9ec932841348b616 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a7d244dad1440f9ec932841348b616" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 37307 } }
I20260926 16:48:38.521466 27417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61a7d244dad1440f9ec932841348b616 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61a7d244dad1440f9ec932841348b616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a7d244dad1440f9ec932841348b616" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 37307 } } }
I20260926 16:48:38.522018 27417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61a7d244dad1440f9ec932841348b616 [sys.catalog]: This master's current role is: LEADER
I20260926 16:48:38.521596 27418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61a7d244dad1440f9ec932841348b616 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61a7d244dad1440f9ec932841348b616. Latest consensus state: current_term: 1 leader_uuid: "61a7d244dad1440f9ec932841348b616" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a7d244dad1440f9ec932841348b616" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 37307 } } }
I20260926 16:48:38.522632 27418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61a7d244dad1440f9ec932841348b616 [sys.catalog]: This master's current role is: LEADER
I20260926 16:48:38.524224 27425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:48:38.531827 27425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:48:38.544237 27425 catalog_manager.cc:1347] Generated new cluster ID: 8076de85516a4286b0b34a449793773f
I20260926 16:48:38.544405 27425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:48:38.586309 27425 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:48:38.588114 27425 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:48:38.603060 27425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61a7d244dad1440f9ec932841348b616: Generated new TSK 0
I20260926 16:48:38.604643 27425 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
I20260926 16:48:38.677724 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:0
--local_ip_for_outbound_sockets=127.9.238.193
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:37307
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:48:39.049599 27437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:48:39.049947 27437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:48:39.050120 27437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:48:39.057766 27437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:48:39.057938 27437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:48:39.058126 27437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:48:39.068464 27437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:37307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:48:39.069868 27437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:48:39.071209 27437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:48:39.085902 27442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:39.085958 27443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:39.086508 27445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:48:39.087962 27437 server_base.cc:1034] running on GCE node
I20260926 16:48:39.088899 27437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:48:39.090338 27437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:48:39.091743 27437 hybrid_clock.cc:648] HybridClock initialized: now 1790441319091620 us; error 76 us; skew 500 ppm
I20260926 16:48:39.092209 27437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:48:39 dist-test-slave-wt2j krb5kdc[27331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441319, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:48:39.096765 27437 init.cc:377] Logged in from keytab as kudu/127.9.238.193@KRBTEST.COM (short username kudu)
I20260926 16:48:39.098975 27437 webserver.cc:460] Webserver started at http://127.9.238.193:39003/ using document root <none> and password file <none>
I20260926 16:48:39.100212 27437 fs_manager.cc:359] Metadata directory not provided
I20260926 16:48:39.100422 27437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:48:39.100832 27437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:48:39.105008 27437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-0/data/instance:
uuid: "0a79cef350a74dfea296d891105b3718"
format_stamp: "Formatted at 2026-09-26 16:48:39 on dist-test-slave-wt2j"
server_key: "71f458c407734ccf5468054185fb5180"
server_key_iv: "8a6466506e89d7bae34a3c345708cfd4"
server_key_version: "encryptionkey@0"
I20260926 16:48:39.106079 27437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance:
uuid: "0a79cef350a74dfea296d891105b3718"
format_stamp: "Formatted at 2026-09-26 16:48:39 on dist-test-slave-wt2j"
server_key: "71f458c407734ccf5468054185fb5180"
server_key_iv: "8a6466506e89d7bae34a3c345708cfd4"
server_key_version: "encryptionkey@0"
I20260926 16:48:39.113306 27437 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260926 16:48:39.118101 27452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:39.119952 27437 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 16:48:39.120143 27437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "0a79cef350a74dfea296d891105b3718"
format_stamp: "Formatted at 2026-09-26 16:48:39 on dist-test-slave-wt2j"
server_key: "71f458c407734ccf5468054185fb5180"
server_key_iv: "8a6466506e89d7bae34a3c345708cfd4"
server_key_version: "encryptionkey@0"
I20260926 16:48:39.120415 27437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:48:39.149385 27437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:48:39.155829 27437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:48:39.156198 27437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:48:39.157501 27437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:48:39.159333 27437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:48:39.159428 27437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:39.159538 27437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:48:39.159672 27437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260926 16:48:39.202342 27437 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:44751
I20260926 16:48:39.202401 27564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:44751 every 8 connection(s)
I20260926 16:48:39.204109 27437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:48:39.213734 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 27437
I20260926 16:48:39.214025 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance
I20260926 16:48:39.214800 10171 external_mini_cluster.cc:1428] Setting key 5bde72ee2d5966e57e422f6bafd17baa
Sep 26 16:48:39 dist-test-slave-wt2j krb5kdc[27331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441319, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:48:39.229184 27567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:39.207198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:43577 (local address 127.9.238.254:37307)
0926 16:48:39.207944 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:39.207954 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:39.208672 (+ 718us) server_negotiation.cc:407] Connection header received
0926 16:48:39.210873 (+ 2201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:39.210879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:39.210987 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:39.211122 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:39.213068 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:39.213973 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:39.215801 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:39.216390 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:39.216539 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:39.221576 (+ 5037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:39.221582 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:39.221586 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:48:39.221624 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:39.224812 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:39.226072 (+ 1260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:39.226082 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:39.226088 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:39.226260 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:39.226755 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:39.226762 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:39.226766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:39.227207 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:39.227387 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:39.228271 (+ 884us) server_negotiation.cc:299] Negotiation successful
0926 16:48:39.228603 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":340,"threads_started":1}
I20260926 16:48:39.229732 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:39.207492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:37307 (local address 127.9.238.193:43577)
0926 16:48:39.208464 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:39.208523 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:39.210545 (+ 2022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:39.211417 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:39.211426 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:39.212118 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:48:39.212881 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:39.212903 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:39.214135 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:39.214143 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:39.215647 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:39.215660 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:39.216678 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:39.216687 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:39.216958 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:39.217882 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:48:39.217908 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:48:39.221258 (+ 3350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:48:39.224991 (+ 3733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:48:39.225005 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:48:39.225028 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:48:39.225804 (+ 776us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:48:39.226383 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:48:39.226386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:48:39.226389 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:48:39.226618 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:48:39.227338 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:48:39.227344 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:48:39.227824 (+ 480us) client_negotiation.cc:769] Sending connection context
0926 16:48:39.228069 (+ 245us) client_negotiation.cc:240] Negotiation successful
0926 16:48:39.228518 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":361,"threads_started":1}
I20260926 16:48:39.232004 27565 heartbeater.cc:347] Connected to a master server at 127.9.238.254:37307
I20260926 16:48:39.232380 27565 heartbeater.cc:464] Registering TS with master...
I20260926 16:48:39.233172 27565 heartbeater.cc:511] Master 127.9.238.254:37307 requested a full tablet report, sending...
I20260926 16:48:39.235982 27376 ts_manager.cc:194] Registered new tserver with Master: 0a79cef350a74dfea296d891105b3718 (127.9.238.193:44751)
I20260926 16:48:39.238444 27376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.193@KRBTEST.COM'} at 127.9.238.193:43577
WARNING: no policy specified for kudu/127.9.238.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
I20260926 16:48:39.283165 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:0
--local_ip_for_outbound_sockets=127.9.238.194
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:37307
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:48:39.680533 27571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:48:39.680972 27571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:48:39.681098 27571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:48:39.688546 27571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:48:39.688673 27571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:48:39.688791 27571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:48:39.698776 27571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:37307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:48:39.700187 27571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:48:39.701644 27571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:48:39.716449 27576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:39.716385 27577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:39.716590 27579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:48:39.719338 27571 server_base.cc:1034] running on GCE node
I20260926 16:48:39.720167 27571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:48:39.721477 27571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:48:39.722721 27571 hybrid_clock.cc:648] HybridClock initialized: now 1790441319722651 us; error 57 us; skew 500 ppm
I20260926 16:48:39.723210 27571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:48:39 dist-test-slave-wt2j krb5kdc[27331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441319, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:48:39.727284 27571 init.cc:377] Logged in from keytab as kudu/127.9.238.194@KRBTEST.COM (short username kudu)
I20260926 16:48:39.729167 27571 webserver.cc:460] Webserver started at http://127.9.238.194:32813/ using document root <none> and password file <none>
I20260926 16:48:39.730118 27571 fs_manager.cc:359] Metadata directory not provided
I20260926 16:48:39.730279 27571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:48:39.730616 27571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:48:39.733922 27571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-1/data/instance:
uuid: "5411842cf8f445cc9cc761afc841338a"
format_stamp: "Formatted at 2026-09-26 16:48:39 on dist-test-slave-wt2j"
server_key: "c241c4e13ce9d9ab28345d3a2e53636e"
server_key_iv: "b957b46471ced56ae2c0ced872379c4a"
server_key_version: "encryptionkey@0"
I20260926 16:48:39.734802 27571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance:
uuid: "5411842cf8f445cc9cc761afc841338a"
format_stamp: "Formatted at 2026-09-26 16:48:39 on dist-test-slave-wt2j"
server_key: "c241c4e13ce9d9ab28345d3a2e53636e"
server_key_iv: "b957b46471ced56ae2c0ced872379c4a"
server_key_version: "encryptionkey@0"
I20260926 16:48:39.741086 27571 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 16:48:39.745520 27586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:39.747357 27571 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:48:39.747660 27571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "5411842cf8f445cc9cc761afc841338a"
format_stamp: "Formatted at 2026-09-26 16:48:39 on dist-test-slave-wt2j"
server_key: "c241c4e13ce9d9ab28345d3a2e53636e"
server_key_iv: "b957b46471ced56ae2c0ced872379c4a"
server_key_version: "encryptionkey@0"
I20260926 16:48:39.747896 27571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:48:39.765234 27571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:48:39.769742 27571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:48:39.770043 27571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:48:39.771133 27571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:48:39.772781 27571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:48:39.772882 27571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:39.773039 27571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:48:39.773144 27571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:39.813601 27571 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:39731
I20260926 16:48:39.813848 27698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:39731 every 8 connection(s)
I20260926 16:48:39.815160 27571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:48:39.824100 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 27571
I20260926 16:48:39.824373 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance
I20260926 16:48:39.825030 10171 external_mini_cluster.cc:1428] Setting key e86beecb16c3f381021e771004794944
Sep 26 16:48:39 dist-test-slave-wt2j krb5kdc[27331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441319, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:48:39.845046 27701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:39.818658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:44937 (local address 127.9.238.254:37307)
0926 16:48:39.819329 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:39.819335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:39.820486 (+ 1151us) server_negotiation.cc:407] Connection header received
0926 16:48:39.822771 (+ 2285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:39.822777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:39.822901 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:39.823038 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:39.825990 (+ 2952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:39.827017 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:39.830294 (+ 3277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:39.830784 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:39.830913 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:39.837676 (+ 6763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:39.837683 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:39.837687 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:48:39.837736 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:39.840800 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:39.841856 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:39.841861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:39.841867 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:39.842053 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:39.842542 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:39.842550 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:39.842556 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:39.842998 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:39.843169 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:39.844214 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0926 16:48:39.844519 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":294,"threads_started":1}
I20260926 16:48:39.846148 27702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:39.819275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:37307 (local address 127.9.238.194:44937)
0926 16:48:39.820237 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:39.820320 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:39.822395 (+ 2075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:39.823391 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:39.823402 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:39.824422 (+ 1020us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:48:39.825542 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:39.825607 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:39.827252 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:39.827258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:39.829937 (+ 2679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:39.830062 (+ 125us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:39.831139 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:39.831149 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:39.831615 (+ 466us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:39.832648 (+ 1033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:48:39.832690 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:48:39.837206 (+ 4516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:48:39.841055 (+ 3849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:48:39.841066 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:48:39.841085 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:48:39.841670 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:48:39.842184 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:48:39.842188 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:48:39.842190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:48:39.842403 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:48:39.843154 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:48:39.843163 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:48:39.843736 (+ 573us) client_negotiation.cc:769] Sending connection context
0926 16:48:39.844067 (+ 331us) client_negotiation.cc:240] Negotiation successful
0926 16:48:39.844626 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":418,"threads_started":1}
I20260926 16:48:39.848151 27699 heartbeater.cc:347] Connected to a master server at 127.9.238.254:37307
I20260926 16:48:39.848546 27699 heartbeater.cc:464] Registering TS with master...
I20260926 16:48:39.849560 27699 heartbeater.cc:511] Master 127.9.238.254:37307 requested a full tablet report, sending...
I20260926 16:48:39.851662 27376 ts_manager.cc:194] Registered new tserver with Master: 5411842cf8f445cc9cc761afc841338a (127.9.238.194:39731)
I20260926 16:48:39.852770 27376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.194@KRBTEST.COM'} at 127.9.238.194:44937
WARNING: no policy specified for kudu/127.9.238.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
I20260926 16:48:39.897256 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:0
--local_ip_for_outbound_sockets=127.9.238.195
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:37307
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:48:40.242650 27565 heartbeater.cc:503] Master 127.9.238.254:37307 was elected leader, sending a full tablet report...
W20260926 16:48:40.262172 27705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:48:40.262562 27705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:48:40.262751 27705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:48:40.270339 27705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:48:40.270550 27705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:48:40.270744 27705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:48:40.280851 27705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:37307
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:48:40.282248 27705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:48:40.283620 27705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:48:40.298199 27711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:40.298324 27710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:40.298406 27713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:48:40.299674 27705 server_base.cc:1034] running on GCE node
I20260926 16:48:40.300443 27705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:48:40.301548 27705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:48:40.302820 27705 hybrid_clock.cc:648] HybridClock initialized: now 1790441320302736 us; error 68 us; skew 500 ppm
I20260926 16:48:40.303210 27705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:48:40 dist-test-slave-wt2j krb5kdc[27331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441320, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:48:40.306603 27705 init.cc:377] Logged in from keytab as kudu/127.9.238.195@KRBTEST.COM (short username kudu)
I20260926 16:48:40.308354 27705 webserver.cc:460] Webserver started at http://127.9.238.195:37673/ using document root <none> and password file <none>
I20260926 16:48:40.309051 27705 fs_manager.cc:359] Metadata directory not provided
I20260926 16:48:40.309175 27705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:48:40.309453 27705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:48:40.312466 27705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-2/data/instance:
uuid: "2edda989a14f498da4ee5dabfe6c9344"
format_stamp: "Formatted at 2026-09-26 16:48:40 on dist-test-slave-wt2j"
server_key: "69d5f8e19518fb529daab42b4adb8508"
server_key_iv: "f7819fa405edd092dca3ddf3da0ccde9"
server_key_version: "encryptionkey@0"
I20260926 16:48:40.313298 27705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance:
uuid: "2edda989a14f498da4ee5dabfe6c9344"
format_stamp: "Formatted at 2026-09-26 16:48:40 on dist-test-slave-wt2j"
server_key: "69d5f8e19518fb529daab42b4adb8508"
server_key_iv: "f7819fa405edd092dca3ddf3da0ccde9"
server_key_version: "encryptionkey@0"
I20260926 16:48:40.319360 27705 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 16:48:40.324051 27720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:40.326033 27705 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:48:40.326220 27705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "2edda989a14f498da4ee5dabfe6c9344"
format_stamp: "Formatted at 2026-09-26 16:48:40 on dist-test-slave-wt2j"
server_key: "69d5f8e19518fb529daab42b4adb8508"
server_key_iv: "f7819fa405edd092dca3ddf3da0ccde9"
server_key_version: "encryptionkey@0"
I20260926 16:48:40.326404 27705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:48:40.344084 27705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:48:40.348832 27705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:48:40.349090 27705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:48:40.350184 27705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:48:40.351935 27705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:48:40.352041 27705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:40.352203 27705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:48:40.352274 27705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:40.391101 27705 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:35471
I20260926 16:48:40.391199 27832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:35471 every 8 connection(s)
I20260926 16:48:40.392504 27705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:48:40.398474 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 27705
I20260926 16:48:40.398720 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance
I20260926 16:48:40.399363 10171 external_mini_cluster.cc:1428] Setting key 43ffd2cbbf32d178b7809e0160f1af22
Sep 26 16:48:40 dist-test-slave-wt2j krb5kdc[27331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441320, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:48:40.417423 27835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:40.395754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:43611 (local address 127.9.238.254:37307)
0926 16:48:40.396275 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:40.396301 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:40.397397 (+ 1096us) server_negotiation.cc:407] Connection header received
0926 16:48:40.400403 (+ 3006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:40.400409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:40.400549 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:40.400702 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:40.402725 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.403866 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.405585 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.406031 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.406139 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:40.410155 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:40.410168 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:40.410172 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:48:40.410291 (+ 119us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:40.413521 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:40.414600 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:40.414607 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:40.414610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:40.414773 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:40.415277 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:40.415283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:40.415286 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:40.415741 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:40.415920 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:40.416836 (+ 916us) server_negotiation.cc:299] Negotiation successful
0926 16:48:40.417083 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":233,"threads_started":1}
I20260926 16:48:40.418488 27836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:40.396331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:37307 (local address 127.9.238.195:43611)
0926 16:48:40.397182 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:40.397233 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:40.399986 (+ 2753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:40.401145 (+ 1159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:40.401165 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:40.401810 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:48:40.402527 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:40.402547 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.404084 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.404088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:40.405425 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:40.405439 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.406273 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.406278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:40.406489 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:40.407270 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:48:40.407293 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:48:40.409853 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:48:40.413754 (+ 3901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:48:40.413763 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:48:40.413778 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:48:40.414433 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:48:40.414893 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:48:40.414900 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:48:40.414904 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:48:40.415162 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:48:40.415871 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:48:40.415879 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:48:40.416328 (+ 449us) client_negotiation.cc:769] Sending connection context
0926 16:48:40.416694 (+ 366us) client_negotiation.cc:240] Negotiation successful
0926 16:48:40.417141 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":307,"threads_started":1}
I20260926 16:48:40.420060 27833 heartbeater.cc:347] Connected to a master server at 127.9.238.254:37307
I20260926 16:48:40.420447 27833 heartbeater.cc:464] Registering TS with master...
I20260926 16:48:40.421360 27833 heartbeater.cc:511] Master 127.9.238.254:37307 requested a full tablet report, sending...
I20260926 16:48:40.423184 27376 ts_manager.cc:194] Registered new tserver with Master: 2edda989a14f498da4ee5dabfe6c9344 (127.9.238.195:35471)
I20260926 16:48:40.424218 27376 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.195@KRBTEST.COM'} at 127.9.238.195:43611
I20260926 16:48:40.429461 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:48:40.449528 27835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:40.435199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40616 (local address 127.9.238.254:37307)
0926 16:48:40.435450 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:40.435454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:40.435895 (+ 441us) server_negotiation.cc:407] Connection header received
0926 16:48:40.436118 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:40.436123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:40.436211 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:40.436321 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:40.437929 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.438707 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.440291 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.440715 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.440827 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:40.443653 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:40.443660 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:40.443665 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:48:40.443709 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:40.446671 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:40.447424 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:40.447428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:40.447431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:40.447631 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:40.448068 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:40.448071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:40.448073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:40.448414 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:40.448603 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:40.449082 (+ 479us) server_negotiation.cc:299] Negotiation successful
0926 16:48:40.449275 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260926 16:48:40.454753 27376 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:48:40.463199 27376 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:48:40.503532 27849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:40.479169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:44751 (local address 127.0.0.1:59040)
0926 16:48:40.490718 (+ 11549us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:40.490750 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:40.492554 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:40.495377 (+ 2823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:40.495384 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:40.495419 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:48:40.496031 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:40.496067 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.497812 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.497819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:40.500439 (+ 2620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:40.500459 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.502408 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.502414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:40.502625 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:40.502690 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 16:48:40.502806 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 16:48:40.503005 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11342,"mutex_wait_us":40,"thread_start_us":13025,"threads_started":1}
I20260926 16:48:40.504490 27851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:40.492162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:39731 (local address 127.0.0.1:43594)
0926 16:48:40.493900 (+ 1738us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:40.493931 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:40.494811 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:40.496400 (+ 1589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:40.496405 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:40.496433 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:48:40.496846 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:40.496860 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.498692 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.498699 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:40.500772 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:40.500791 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.503270 (+ 2479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.503277 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:40.503500 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:40.503566 (+ 66us) client_negotiation.cc:769] Sending connection context
0926 16:48:40.503674 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 16:48:40.503851 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1530,"mutex_wait_us":25,"thread_start_us":1320,"threads_started":1}
I20260926 16:48:40.504828 27846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:40.478290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:35471 (local address 127.0.0.1:50532)
0926 16:48:40.486013 (+ 7723us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:40.486063 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:40.486296 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:40.495620 (+ 9324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:40.495625 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:40.495658 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:48:40.496251 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:40.496269 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.498239 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.498246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:40.501839 (+ 3593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:40.501857 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.503943 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.503950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:40.504155 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:40.504192 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 16:48:40.504266 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 16:48:40.504421 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7390,"mutex_wait_us":56,"thread_start_us":7211,"threads_started":1}
I20260926 16:48:40.505024 27848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:40.479387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59040 (local address 127.9.238.193:44751)
0926 16:48:40.493368 (+ 13981us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:40.493393 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:40.493437 (+ 44us) server_negotiation.cc:407] Connection header received
0926 16:48:40.493576 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:40.493586 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:40.494024 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:40.496027 (+ 2003us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:48:40.496300 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.497667 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.500668 (+ 3001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.502192 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.504118 (+ 1926us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:40.504381 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:40.504452 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 16:48:40.504645 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13713,"thread_start_us":13511,"threads_started":1}
I20260926 16:48:40.505470 27850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:40.487070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43594 (local address 127.9.238.194:39731)
0926 16:48:40.495419 (+ 8349us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:40.495429 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:40.495461 (+ 32us) server_negotiation.cc:407] Connection header received
0926 16:48:40.495633 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:40.495645 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:40.495950 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:40.496158 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:48:40.497070 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.498526 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.501003 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.502643 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.502808 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:40.503069 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:40.504823 (+ 1754us) server_negotiation.cc:299] Negotiation successful
0926 16:48:40.505073 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8150,"thread_start_us":7986,"threads_started":1}
I20260926 16:48:40.506017 27847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:40.478686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50532 (local address 127.9.238.195:35471)
0926 16:48:40.494947 (+ 16261us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:40.494957 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:40.494997 (+ 40us) server_negotiation.cc:407] Connection header received
0926 16:48:40.495121 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:40.495129 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:40.495342 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:40.495620 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:48:40.496463 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.498068 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.502087 (+ 4019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.503669 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.505148 (+ 1479us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:40.505384 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:40.505477 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 16:48:40.505650 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16021,"thread_start_us":15783,"threads_started":1}
I20260926 16:48:40.509150 27500 tablet_service.cc:1467] Processing CreateTablet for tablet 4a5ee078004a448499ebfe84d17ee7e2 (DEFAULT_TABLE table=test-table [id=94873a12672544029f169178db9ec473]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:48:40.509531 27634 tablet_service.cc:1467] Processing CreateTablet for tablet 4a5ee078004a448499ebfe84d17ee7e2 (DEFAULT_TABLE table=test-table [id=94873a12672544029f169178db9ec473]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:48:40.510800 27634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a5ee078004a448499ebfe84d17ee7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:48:40.510797 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a5ee078004a448499ebfe84d17ee7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:48:40.511667 27768 tablet_service.cc:1467] Processing CreateTablet for tablet 4a5ee078004a448499ebfe84d17ee7e2 (DEFAULT_TABLE table=test-table [id=94873a12672544029f169178db9ec473]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:48:40.513427 27768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a5ee078004a448499ebfe84d17ee7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:48:40.526687 27852 tablet_bootstrap.cc:492] T 4a5ee078004a448499ebfe84d17ee7e2 P 0a79cef350a74dfea296d891105b3718: Bootstrap starting.
I20260926 16:48:40.528544 27853 tablet_bootstrap.cc:492] T 4a5ee078004a448499ebfe84d17ee7e2 P 5411842cf8f445cc9cc761afc841338a: Bootstrap starting.
I20260926 16:48:40.529233 27854 tablet_bootstrap.cc:492] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344: Bootstrap starting.
I20260926 16:48:40.532704 27853 tablet_bootstrap.cc:654] T 4a5ee078004a448499ebfe84d17ee7e2 P 5411842cf8f445cc9cc761afc841338a: Neither blocks nor log segments found. Creating new log.
I20260926 16:48:40.532784 27854 tablet_bootstrap.cc:654] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344: Neither blocks nor log segments found. Creating new log.
I20260926 16:48:40.534092 27854 log.cc:824] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344: Log is configured to *not* fsync() on all Append() calls
I20260926 16:48:40.534092 27853 log.cc:824] T 4a5ee078004a448499ebfe84d17ee7e2 P 5411842cf8f445cc9cc761afc841338a: Log is configured to *not* fsync() on all Append() calls
I20260926 16:48:40.534301 27852 tablet_bootstrap.cc:654] T 4a5ee078004a448499ebfe84d17ee7e2 P 0a79cef350a74dfea296d891105b3718: Neither blocks nor log segments found. Creating new log.
I20260926 16:48:40.536125 27852 log.cc:824] T 4a5ee078004a448499ebfe84d17ee7e2 P 0a79cef350a74dfea296d891105b3718: Log is configured to *not* fsync() on all Append() calls
I20260926 16:48:40.538117 27853 tablet_bootstrap.cc:492] T 4a5ee078004a448499ebfe84d17ee7e2 P 5411842cf8f445cc9cc761afc841338a: No bootstrap required, opened a new log
I20260926 16:48:40.538257 27854 tablet_bootstrap.cc:492] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344: No bootstrap required, opened a new log
I20260926 16:48:40.538491 27853 ts_tablet_manager.cc:1397] T 4a5ee078004a448499ebfe84d17ee7e2 P 5411842cf8f445cc9cc761afc841338a: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260926 16:48:40.538653 27854 ts_tablet_manager.cc:1397] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260926 16:48:40.539674 27852 tablet_bootstrap.cc:492] T 4a5ee078004a448499ebfe84d17ee7e2 P 0a79cef350a74dfea296d891105b3718: No bootstrap required, opened a new log
I20260926 16:48:40.539975 27852 ts_tablet_manager.cc:1397] T 4a5ee078004a448499ebfe84d17ee7e2 P 0a79cef350a74dfea296d891105b3718: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20260926 16:48:40.546515 27853 raft_consensus.cc:348] T 4a5ee078004a448499ebfe84d17ee7e2 P 5411842cf8f445cc9cc761afc841338a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2edda989a14f498da4ee5dabfe6c9344" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35471 } } peers { permanent_uuid: "0a79cef350a74dfea296d891105b3718" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44751 } } peers { permanent_uuid: "5411842cf8f445cc9cc761afc841338a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 39731 } }
I20260926 16:48:40.546933 27853 raft_consensus.cc:374] T 4a5ee078004a448499ebfe84d17ee7e2 P 5411842cf8f445cc9cc761afc841338a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:48:40.547039 27853 raft_consensus.cc:729] T 4a5ee078004a448499ebfe84d17ee7e2 P 5411842cf8f445cc9cc761afc841338a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5411842cf8f445cc9cc761afc841338a, State: Initialized, Role: FOLLOWER
I20260926 16:48:40.547737 27853 consensus_queue.cc:260] T 4a5ee078004a448499ebfe84d17ee7e2 P 5411842cf8f445cc9cc761afc841338a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2edda989a14f498da4ee5dabfe6c9344" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35471 } } peers { permanent_uuid: "0a79cef350a74dfea296d891105b3718" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44751 } } peers { permanent_uuid: "5411842cf8f445cc9cc761afc841338a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 39731 } }
I20260926 16:48:40.549093 27699 heartbeater.cc:503] Master 127.9.238.254:37307 was elected leader, sending a full tablet report...
I20260926 16:48:40.549512 27853 ts_tablet_manager.cc:1428] T 4a5ee078004a448499ebfe84d17ee7e2 P 5411842cf8f445cc9cc761afc841338a: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260926 16:48:40.550954 27854 raft_consensus.cc:348] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2edda989a14f498da4ee5dabfe6c9344" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35471 } } peers { permanent_uuid: "0a79cef350a74dfea296d891105b3718" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44751 } } peers { permanent_uuid: "5411842cf8f445cc9cc761afc841338a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 39731 } }
I20260926 16:48:40.551625 27854 raft_consensus.cc:374] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:48:40.551780 27854 raft_consensus.cc:729] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edda989a14f498da4ee5dabfe6c9344, State: Initialized, Role: FOLLOWER
I20260926 16:48:40.552124 27852 raft_consensus.cc:348] T 4a5ee078004a448499ebfe84d17ee7e2 P 0a79cef350a74dfea296d891105b3718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2edda989a14f498da4ee5dabfe6c9344" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35471 } } peers { permanent_uuid: "0a79cef350a74dfea296d891105b3718" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44751 } } peers { permanent_uuid: "5411842cf8f445cc9cc761afc841338a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 39731 } }
I20260926 16:48:40.552590 27852 raft_consensus.cc:374] T 4a5ee078004a448499ebfe84d17ee7e2 P 0a79cef350a74dfea296d891105b3718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:48:40.552691 27852 raft_consensus.cc:729] T 4a5ee078004a448499ebfe84d17ee7e2 P 0a79cef350a74dfea296d891105b3718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a79cef350a74dfea296d891105b3718, State: Initialized, Role: FOLLOWER
I20260926 16:48:40.552570 27854 consensus_queue.cc:260] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2edda989a14f498da4ee5dabfe6c9344" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35471 } } peers { permanent_uuid: "0a79cef350a74dfea296d891105b3718" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44751 } } peers { permanent_uuid: "5411842cf8f445cc9cc761afc841338a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 39731 } }
I20260926 16:48:40.553319 27852 consensus_queue.cc:260] T 4a5ee078004a448499ebfe84d17ee7e2 P 0a79cef350a74dfea296d891105b3718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2edda989a14f498da4ee5dabfe6c9344" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35471 } } peers { permanent_uuid: "0a79cef350a74dfea296d891105b3718" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44751 } } peers { permanent_uuid: "5411842cf8f445cc9cc761afc841338a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 39731 } }
I20260926 16:48:40.554035 27833 heartbeater.cc:503] Master 127.9.238.254:37307 was elected leader, sending a full tablet report...
I20260926 16:48:40.555383 27854 ts_tablet_manager.cc:1428] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344: Time spent starting tablet: real 0.017s user 0.013s sys 0.006s
I20260926 16:48:40.559821 27852 ts_tablet_manager.cc:1428] T 4a5ee078004a448499ebfe84d17ee7e2 P 0a79cef350a74dfea296d891105b3718: Time spent starting tablet: real 0.020s user 0.015s sys 0.004s
W20260926 16:48:40.567617 27700 tablet.cc:2369] T 4a5ee078004a448499ebfe84d17ee7e2 P 5411842cf8f445cc9cc761afc841338a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:48:40.644426 27834 tablet.cc:2369] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:48:40.707521 27566 tablet.cc:2369] T 4a5ee078004a448499ebfe84d17ee7e2 P 0a79cef350a74dfea296d891105b3718: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:48:40.723239 27859 raft_consensus.cc:482] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:48:40.723579 27859 raft_consensus.cc:504] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2edda989a14f498da4ee5dabfe6c9344" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35471 } } peers { permanent_uuid: "0a79cef350a74dfea296d891105b3718" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44751 } } peers { permanent_uuid: "5411842cf8f445cc9cc761afc841338a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 39731 } }
I20260926 16:48:40.725462 27859 leader_election.cc:288] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a79cef350a74dfea296d891105b3718 (127.9.238.193:44751), 5411842cf8f445cc9cc761afc841338a (127.9.238.194:39731)
I20260926 16:48:40.733346 27836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:40.725779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:39731 (local address 127.9.238.195:41475)
0926 16:48:40.726010 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:40.726035 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:40.726179 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:40.726896 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:40.726903 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:40.726960 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:48:40.727345 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:40.727363 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.728866 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.728873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:40.730891 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:40.730906 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.732496 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.732503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:40.732739 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:40.732771 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 16:48:40.732878 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 16:48:40.733042 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 16:48:40.733348 27850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:40.726277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:41475 (local address 127.9.238.194:39731)
0926 16:48:40.726477 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:40.726483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:40.726505 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:48:40.726585 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:40.726593 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:40.726681 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:40.726792 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:48:40.727688 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.728705 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.731186 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.732327 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.732509 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:40.732585 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:40.732905 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 16:48:40.733062 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:48:40.733920 27848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:40.726221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:39785 (local address 127.9.238.193:44751)
0926 16:48:40.726499 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:40.726506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:40.727182 (+ 676us) server_negotiation.cc:407] Connection header received
0926 16:48:40.727302 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:40.727309 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:40.727408 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:40.727587 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:48:40.728366 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.729509 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.731811 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.732807 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.733027 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:40.733117 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:40.733447 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 16:48:40.733629 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260926 16:48:40.734457 27861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:40.726216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:44751 (local address 127.9.238.195:39785)
0926 16:48:40.727059 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:40.727074 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:40.727179 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:40.727694 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:40.727699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:40.727730 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:48:40.728139 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:40.728154 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.729687 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.729691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:40.731628 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:40.731645 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.733052 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.733059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:40.733256 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:40.733283 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 16:48:40.733409 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 16:48:40.734137 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":430,"threads_started":1}
I20260926 16:48:40.734359 27654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a5ee078004a448499ebfe84d17ee7e2" candidate_uuid: "2edda989a14f498da4ee5dabfe6c9344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5411842cf8f445cc9cc761afc841338a" is_pre_election: true
I20260926 16:48:40.734894 27654 raft_consensus.cc:2454] T 4a5ee078004a448499ebfe84d17ee7e2 P 5411842cf8f445cc9cc761afc841338a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2edda989a14f498da4ee5dabfe6c9344 in term 0.
I20260926 16:48:40.735612 27724 leader_election.cc:302] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edda989a14f498da4ee5dabfe6c9344, 5411842cf8f445cc9cc761afc841338a; no voters:
I20260926 16:48:40.735761 27520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a5ee078004a448499ebfe84d17ee7e2" candidate_uuid: "2edda989a14f498da4ee5dabfe6c9344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a79cef350a74dfea296d891105b3718" is_pre_election: true
I20260926 16:48:40.736039 27859 raft_consensus.cc:2781] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:48:40.736204 27859 raft_consensus.cc:482] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:48:40.736310 27859 raft_consensus.cc:3037] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:48:40.736509 27520 raft_consensus.cc:2454] T 4a5ee078004a448499ebfe84d17ee7e2 P 0a79cef350a74dfea296d891105b3718 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2edda989a14f498da4ee5dabfe6c9344 in term 0.
I20260926 16:48:40.738773 27859 raft_consensus.cc:504] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2edda989a14f498da4ee5dabfe6c9344" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35471 } } peers { permanent_uuid: "0a79cef350a74dfea296d891105b3718" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44751 } } peers { permanent_uuid: "5411842cf8f445cc9cc761afc841338a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 39731 } }
I20260926 16:48:40.739480 27859 leader_election.cc:288] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344 [CANDIDATE]: Term 1 election: Requested vote from peers 0a79cef350a74dfea296d891105b3718 (127.9.238.193:44751), 5411842cf8f445cc9cc761afc841338a (127.9.238.194:39731)
I20260926 16:48:40.740005 27520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a5ee078004a448499ebfe84d17ee7e2" candidate_uuid: "2edda989a14f498da4ee5dabfe6c9344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a79cef350a74dfea296d891105b3718"
I20260926 16:48:40.740191 27654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a5ee078004a448499ebfe84d17ee7e2" candidate_uuid: "2edda989a14f498da4ee5dabfe6c9344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5411842cf8f445cc9cc761afc841338a"
I20260926 16:48:40.740281 27520 raft_consensus.cc:3037] T 4a5ee078004a448499ebfe84d17ee7e2 P 0a79cef350a74dfea296d891105b3718 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:48:40.740455 27654 raft_consensus.cc:3037] T 4a5ee078004a448499ebfe84d17ee7e2 P 5411842cf8f445cc9cc761afc841338a [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:48:40.744108 27654 raft_consensus.cc:2454] T 4a5ee078004a448499ebfe84d17ee7e2 P 5411842cf8f445cc9cc761afc841338a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edda989a14f498da4ee5dabfe6c9344 in term 1.
I20260926 16:48:40.744108 27520 raft_consensus.cc:2454] T 4a5ee078004a448499ebfe84d17ee7e2 P 0a79cef350a74dfea296d891105b3718 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edda989a14f498da4ee5dabfe6c9344 in term 1.
I20260926 16:48:40.744670 27723 leader_election.cc:302] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a79cef350a74dfea296d891105b3718, 2edda989a14f498da4ee5dabfe6c9344; no voters:
I20260926 16:48:40.745062 27859 raft_consensus.cc:2781] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:48:40.745631 27859 raft_consensus.cc:686] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344 [term 1 LEADER]: Becoming Leader. State: Replica: 2edda989a14f498da4ee5dabfe6c9344, State: Running, Role: LEADER
I20260926 16:48:40.746196 27859 consensus_queue.cc:237] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2edda989a14f498da4ee5dabfe6c9344" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35471 } } peers { permanent_uuid: "0a79cef350a74dfea296d891105b3718" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44751 } } peers { permanent_uuid: "5411842cf8f445cc9cc761afc841338a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 39731 } }
I20260926 16:48:40.751029 27376 catalog_manager.cc:5495] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2edda989a14f498da4ee5dabfe6c9344 (127.9.238.195). New cstate: current_term: 1 leader_uuid: "2edda989a14f498da4ee5dabfe6c9344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2edda989a14f498da4ee5dabfe6c9344" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 35471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a79cef350a74dfea296d891105b3718" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5411842cf8f445cc9cc761afc841338a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 39731 } health_report { overall_health: UNKNOWN } } }
I20260926 16:48:40.791286 27847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:40.785729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50542 (local address 127.9.238.195:35471)
0926 16:48:40.786029 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:40.786036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:40.786059 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:48:40.786245 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:40.786249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:40.786332 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:40.786494 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:48:40.786869 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.787704 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.789104 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.789500 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.789632 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:40.790165 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:48:40.790689 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:48:40.790854 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:40.790910 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 16:48:40.790997 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260926 16:48:40.801820 27654 raft_consensus.cc:1261] T 4a5ee078004a448499ebfe84d17ee7e2 P 5411842cf8f445cc9cc761afc841338a [term 1 FOLLOWER]: Refusing update from remote peer 2edda989a14f498da4ee5dabfe6c9344: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:48:40.801875 27520 raft_consensus.cc:1261] T 4a5ee078004a448499ebfe84d17ee7e2 P 0a79cef350a74dfea296d891105b3718 [term 1 FOLLOWER]: Refusing update from remote peer 2edda989a14f498da4ee5dabfe6c9344: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:48:40.803098 27862 consensus_queue.cc:1035] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5411842cf8f445cc9cc761afc841338a" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 39731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:48:40.803593 27859 consensus_queue.cc:1035] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a79cef350a74dfea296d891105b3718" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:48:40.820097 27867 mvcc.cc:205] Tried to move back new op lower bound from 7333647649989500928 to 7333647649779982336. Current Snapshot: MvccSnapshot[applied={T|T < 7333647649989500928}]
I20260926 16:48:40.823017 27865 mvcc.cc:205] Tried to move back new op lower bound from 7333647649989500928 to 7333647649779982336. Current Snapshot: MvccSnapshot[applied={T|T < 7333647649989500928 or (T in {7333647649989500928})}]
I20260926 16:48:40.838739 27375 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:48:40.839079 27375 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:48:40.843994 27375 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61a7d244dad1440f9ec932841348b616: Sending DeleteTablet for 3 replicas of tablet 4a5ee078004a448499ebfe84d17ee7e2
I20260926 16:48:40.845487 27500 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a5ee078004a448499ebfe84d17ee7e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:48:40 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:59040
I20260926 16:48:40.845577 27634 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a5ee078004a448499ebfe84d17ee7e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:48:40 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:43594
I20260926 16:48:40.846366 27768 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a5ee078004a448499ebfe84d17ee7e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:48:40 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:50532
I20260926 16:48:40.847239 27874 tablet_replica.cc:330] T 4a5ee078004a448499ebfe84d17ee7e2 P 0a79cef350a74dfea296d891105b3718: stopping tablet replica
I20260926 16:48:40.847955 27874 raft_consensus.cc:2229] T 4a5ee078004a448499ebfe84d17ee7e2 P 0a79cef350a74dfea296d891105b3718 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:48:40.848069 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 27437
I20260926 16:48:40.848704 27874 raft_consensus.cc:2258] T 4a5ee078004a448499ebfe84d17ee7e2 P 0a79cef350a74dfea296d891105b3718 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:48:40.851929 27876 tablet_replica.cc:330] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344: stopping tablet replica
I20260926 16:48:40.852087 27875 tablet_replica.cc:330] T 4a5ee078004a448499ebfe84d17ee7e2 P 5411842cf8f445cc9cc761afc841338a: stopping tablet replica
I20260926 16:48:40.853053 27875 raft_consensus.cc:2229] T 4a5ee078004a448499ebfe84d17ee7e2 P 5411842cf8f445cc9cc761afc841338a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:48:40.853598 27876 raft_consensus.cc:2229] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:48:40.853724 27875 raft_consensus.cc:2258] T 4a5ee078004a448499ebfe84d17ee7e2 P 5411842cf8f445cc9cc761afc841338a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:48:40.854224 27874 ts_tablet_manager.cc:1905] T 4a5ee078004a448499ebfe84d17ee7e2 P 0a79cef350a74dfea296d891105b3718: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:48:40.854382 27876 raft_consensus.cc:2258] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:48:40.857034 27875 ts_tablet_manager.cc:1905] T 4a5ee078004a448499ebfe84d17ee7e2 P 5411842cf8f445cc9cc761afc841338a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:48:40.862574 27874 ts_tablet_manager.cc:1918] T 4a5ee078004a448499ebfe84d17ee7e2 P 0a79cef350a74dfea296d891105b3718: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:48:40.862828 27874 log.cc:1196] T 4a5ee078004a448499ebfe84d17ee7e2 P 0a79cef350a74dfea296d891105b3718: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-0/wal/wals/4a5ee078004a448499ebfe84d17ee7e2
I20260926 16:48:40.863451 27874 ts_tablet_manager.cc:1939] T 4a5ee078004a448499ebfe84d17ee7e2 P 0a79cef350a74dfea296d891105b3718: Deleting consensus metadata
I20260926 16:48:40.866082 27362 catalog_manager.cc:4841] TS 0a79cef350a74dfea296d891105b3718 (127.9.238.193:44751): tablet 4a5ee078004a448499ebfe84d17ee7e2 (table test-table [id=94873a12672544029f169178db9ec473]) successfully deleted
I20260926 16:48:40.867226 27876 ts_tablet_manager.cc:1905] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:48:40.867769 27875 ts_tablet_manager.cc:1918] T 4a5ee078004a448499ebfe84d17ee7e2 P 5411842cf8f445cc9cc761afc841338a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:48:40.868048 27875 log.cc:1196] T 4a5ee078004a448499ebfe84d17ee7e2 P 5411842cf8f445cc9cc761afc841338a: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-1/wal/wals/4a5ee078004a448499ebfe84d17ee7e2
I20260926 16:48:40.868693 27875 ts_tablet_manager.cc:1939] T 4a5ee078004a448499ebfe84d17ee7e2 P 5411842cf8f445cc9cc761afc841338a: Deleting consensus metadata
I20260926 16:48:40.870860 27363 catalog_manager.cc:4841] TS 5411842cf8f445cc9cc761afc841338a (127.9.238.194:39731): tablet 4a5ee078004a448499ebfe84d17ee7e2 (table test-table [id=94873a12672544029f169178db9ec473]) successfully deleted
I20260926 16:48:40.876258 27876 ts_tablet_manager.cc:1918] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:48:40.876503 27876 log.cc:1196] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790440988577176-10171-0/minicluster-data/ts-2/wal/wals/4a5ee078004a448499ebfe84d17ee7e2
I20260926 16:48:40.877177 27876 ts_tablet_manager.cc:1939] T 4a5ee078004a448499ebfe84d17ee7e2 P 2edda989a14f498da4ee5dabfe6c9344: Deleting consensus metadata
Sep 26 16:48:40 dist-test-slave-wt2j krb5kdc[27331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.193@KRBTEST.COM
I20260926 16:48:40.879244 27363 catalog_manager.cc:4841] TS 2edda989a14f498da4ee5dabfe6c9344 (127.9.238.195:35471): tablet 4a5ee078004a448499ebfe84d17ee7e2 (table test-table [id=94873a12672544029f169178db9ec473]) successfully deleted
I20260926 16:48:40.886838 27848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:40.851567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59056 (local address 127.9.238.193:44751)
0926 16:48:40.851779 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:40.851788 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:40.864666 (+ 12878us) server_negotiation.cc:407] Connection header received
0926 16:48:40.866152 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:40.866157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:40.866254 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:40.866448 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:40.868844 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.870053 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.872212 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:40.872972 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:40.873206 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:40.879632 (+ 6426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:40.879653 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:40.879674 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:48:40.879729 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:40.883208 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:40.884341 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:40.884351 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:40.884355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:40.884531 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:40.885011 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:40.885015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:40.885017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:40.885499 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:40.885763 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:40.886226 (+ 463us) server_negotiation.cc:299] Negotiation successful
0926 16:48:40.886430 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 16:48:40.887701 27560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:48:40.991633 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 27437
I20260926 16:48:41.017434 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 27571
Sep 26 16:48:41 dist-test-slave-wt2j krb5kdc[27331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.194@KRBTEST.COM
I20260926 16:48:41.038039 27850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:41.018622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43596 (local address 127.9.238.194:39731)
0926 16:48:41.018886 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:41.018892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:41.018915 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:48:41.019009 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:41.019014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:41.019097 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:41.019204 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:41.020979 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:41.022104 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:41.024316 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:41.024861 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:41.024992 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:41.030991 (+ 5999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:41.031014 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:41.031036 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:48:41.031091 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:41.034595 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:41.035615 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:41.035628 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:41.035632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:41.035797 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:41.036227 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:41.036237 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:41.036239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:41.036695 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:41.037059 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:41.037359 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 16:48:41.037617 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 16:48:41.038877 27694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:48:41.143023 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 27571
I20260926 16:48:41.165091 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 27705
Sep 26 16:48:41 dist-test-slave-wt2j krb5kdc[27331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.195@KRBTEST.COM
I20260926 16:48:41.187448 27847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:41.166313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48410 (local address 127.9.238.195:35471)
0926 16:48:41.166551 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:41.166556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:41.166595 (+ 39us) server_negotiation.cc:407] Connection header received
0926 16:48:41.166677 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:41.166681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:41.166795 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:41.166889 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:41.168836 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:41.169819 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:41.171937 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:41.172462 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:41.172588 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:41.178781 (+ 6193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:41.178810 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:41.178842 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:48:41.178917 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:41.183655 (+ 4738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:41.184838 (+ 1183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:41.184851 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:41.184856 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:41.185069 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:41.185540 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:41.185545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:41.185549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:41.186076 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:41.186433 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:41.186761 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 16:48:41.187025 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 16:48:41.188517 27828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:48:41.337546 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 27705
I20260926 16:48:41.364128 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 27344
I20260926 16:48:41.366004 27406 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:48:41.482558 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 27344
2026-09-26T16:48:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (5085 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T16:48:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:48:41Z Disabled control of system clock
I20260926 16:48:41.585963 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:34623
--webserver_interface=127.9.238.254
--webserver_port=0
--builtin_ntp_servers=127.9.238.212:38223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:34623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 16:48:41.937561 27885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:48:41.937883 27885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:48:41.938023 27885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:48:41.945713 27885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:48:41.945835 27885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:48:41.945874 27885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:48:41.946012 27885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:48:41.946110 27885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:48:41.956338 27885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:38223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:34623
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:34623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:48:41.957736 27885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:48:41.959859 27885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:48:41.973127 27890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:41.973421 27891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:41.973150 27893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:48:41.975521 27885 server_base.cc:1034] running on GCE node
I20260926 16:48:41.976688 27885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:48:41.978408 27885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:48:41.979672 27885 hybrid_clock.cc:648] HybridClock initialized: now 1790441321979595 us; error 79 us; skew 500 ppm
I20260926 16:48:41.980144 27885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:48:41.983031 27885 webserver.cc:460] Webserver started at http://127.9.238.254:33187/ using document root <none> and password file <none>
I20260926 16:48:41.983909 27885 fs_manager.cc:359] Metadata directory not provided
I20260926 16:48:41.984059 27885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:48:41.984477 27885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:48:41.987721 27885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "3c7999080535453bb1933073ef8fbc58"
format_stamp: "Formatted at 2026-09-26 16:48:41 on dist-test-slave-wt2j"
server_key: "60ce572deab80a1d5d2f57d7bc205781"
server_key_iv: "b10bcb0b70396b0999d8221e216d944d"
server_key_version: "encryptionkey@0"
I20260926 16:48:41.988605 27885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "3c7999080535453bb1933073ef8fbc58"
format_stamp: "Formatted at 2026-09-26 16:48:41 on dist-test-slave-wt2j"
server_key: "60ce572deab80a1d5d2f57d7bc205781"
server_key_iv: "b10bcb0b70396b0999d8221e216d944d"
server_key_version: "encryptionkey@0"
I20260926 16:48:41.994952 27885 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 16:48:41.999810 27899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:42.002725 27885 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260926 16:48:42.003024 27885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "3c7999080535453bb1933073ef8fbc58"
format_stamp: "Formatted at 2026-09-26 16:48:41 on dist-test-slave-wt2j"
server_key: "60ce572deab80a1d5d2f57d7bc205781"
server_key_iv: "b10bcb0b70396b0999d8221e216d944d"
server_key_version: "encryptionkey@0"
I20260926 16:48:42.003386 27885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:48:42.030826 27885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:48:42.031749 27885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:48:42.032018 27885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:48:42.054281 27885 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:34623
I20260926 16:48:42.054405 27950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:34623 every 8 connection(s)
I20260926 16:48:42.055838 27885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:48:42.056531 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 27885
I20260926 16:48:42.056759 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:48:42.057548 10171 external_mini_cluster.cc:1428] Setting key 4ae47d07c092203777057dfd960a7dab
I20260926 16:48:42.060674 27951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:48:42.074885 27951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c7999080535453bb1933073ef8fbc58: Bootstrap starting.
I20260926 16:48:42.076012 27954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:42.060201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33266 (local address 127.9.238.254:34623)
0926 16:48:42.061555 (+ 1354us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:42.061570 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:42.061627 (+ 57us) server_negotiation.cc:407] Connection header received
0926 16:48:42.063784 (+ 2157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:42.063814 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:42.064479 (+ 665us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:42.064936 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:42.065761 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:42.067856 (+ 2095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:42.069911 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:42.070825 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:42.071037 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:42.072621 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:42.072630 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:42.072653 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:48:42.073480 (+ 827us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:42.073592 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:48:42.073618 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:42.073742 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:42.074022 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 16:48:42.074574 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":768,"thread_start_us":500,"threads_started":1}
I20260926 16:48:42.078833 27951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c7999080535453bb1933073ef8fbc58: Neither blocks nor log segments found. Creating new log.
I20260926 16:48:42.080088 27951 log.cc:824] T 00000000000000000000000000000000 P 3c7999080535453bb1933073ef8fbc58: Log is configured to *not* fsync() on all Append() calls
I20260926 16:48:42.083652 27951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c7999080535453bb1933073ef8fbc58: No bootstrap required, opened a new log
I20260926 16:48:42.091178 27951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c7999080535453bb1933073ef8fbc58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7999080535453bb1933073ef8fbc58" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 34623 } }
I20260926 16:48:42.091636 27951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c7999080535453bb1933073ef8fbc58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:48:42.091758 27951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c7999080535453bb1933073ef8fbc58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c7999080535453bb1933073ef8fbc58, State: Initialized, Role: FOLLOWER
I20260926 16:48:42.092422 27951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c7999080535453bb1933073ef8fbc58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7999080535453bb1933073ef8fbc58" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 34623 } }
I20260926 16:48:42.092648 27951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c7999080535453bb1933073ef8fbc58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:48:42.092816 27951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c7999080535453bb1933073ef8fbc58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:48:42.093017 27951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c7999080535453bb1933073ef8fbc58 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:48:42.095664 27951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c7999080535453bb1933073ef8fbc58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7999080535453bb1933073ef8fbc58" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 34623 } }
I20260926 16:48:42.096241 27951 leader_election.cc:302] T 00000000000000000000000000000000 P 3c7999080535453bb1933073ef8fbc58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c7999080535453bb1933073ef8fbc58; no voters:
I20260926 16:48:42.096937 27951 leader_election.cc:288] T 00000000000000000000000000000000 P 3c7999080535453bb1933073ef8fbc58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:48:42.097268 27956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c7999080535453bb1933073ef8fbc58 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:48:42.098326 27956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c7999080535453bb1933073ef8fbc58 [term 1 LEADER]: Becoming Leader. State: Replica: 3c7999080535453bb1933073ef8fbc58, State: Running, Role: LEADER
I20260926 16:48:42.099103 27956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c7999080535453bb1933073ef8fbc58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7999080535453bb1933073ef8fbc58" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 34623 } }
I20260926 16:48:42.099611 27951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c7999080535453bb1933073ef8fbc58 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:48:42.103318 27957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7999080535453bb1933073ef8fbc58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c7999080535453bb1933073ef8fbc58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7999080535453bb1933073ef8fbc58" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 34623 } } }
I20260926 16:48:42.104017 27957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7999080535453bb1933073ef8fbc58 [sys.catalog]: This master's current role is: LEADER
I20260926 16:48:42.103315 27958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7999080535453bb1933073ef8fbc58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c7999080535453bb1933073ef8fbc58. Latest consensus state: current_term: 1 leader_uuid: "3c7999080535453bb1933073ef8fbc58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7999080535453bb1933073ef8fbc58" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 34623 } } }
I20260926 16:48:42.104571 27958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7999080535453bb1933073ef8fbc58 [sys.catalog]: This master's current role is: LEADER
I20260926 16:48:42.107841 27962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:48:42.114872 27962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:48:42.126520 27962 catalog_manager.cc:1347] Generated new cluster ID: 0d33588ca2344c598474ab8d6c385d0d
I20260926 16:48:42.126683 27962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:48:42.156257 27962 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:48:42.158681 27962 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:48:42.170526 27962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c7999080535453bb1933073ef8fbc58: Generated new TSK 0
I20260926 16:48:42.172080 27962 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:48:42.181563 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:0
--local_ip_for_outbound_sockets=127.9.238.193
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:34623
--builtin_ntp_servers=127.9.238.212:38223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 16:48:42.536278 27975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:48:42.536597 27975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:48:42.536741 27975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:48:42.544580 27975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:48:42.544696 27975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:48:42.544868 27975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:48:42.554731 27975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:38223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:34623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:48:42.556380 27975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:48:42.558284 27975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:48:42.574120 27983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:42.574138 27980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:42.575068 27981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:48:42.575130 27975 server_base.cc:1034] running on GCE node
I20260926 16:48:42.576212 27975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:48:42.577504 27975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:48:42.578878 27975 hybrid_clock.cc:648] HybridClock initialized: now 1790441322578786 us; error 87 us; skew 500 ppm
I20260926 16:48:42.579291 27975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:48:42.582522 27975 webserver.cc:460] Webserver started at http://127.9.238.193:38849/ using document root <none> and password file <none>
I20260926 16:48:42.583353 27975 fs_manager.cc:359] Metadata directory not provided
I20260926 16:48:42.583455 27975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:48:42.583884 27975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:48:42.587091 27975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-0/data/instance:
uuid: "5e696c7a6fe3480da0c4091d5eebd052"
format_stamp: "Formatted at 2026-09-26 16:48:42 on dist-test-slave-wt2j"
server_key: "72076bfed53cfe5c4b67d92abbf0ba06"
server_key_iv: "7cdd76683ae6997875116ebaf23bd40a"
server_key_version: "encryptionkey@0"
I20260926 16:48:42.587998 27975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance:
uuid: "5e696c7a6fe3480da0c4091d5eebd052"
format_stamp: "Formatted at 2026-09-26 16:48:42 on dist-test-slave-wt2j"
server_key: "72076bfed53cfe5c4b67d92abbf0ba06"
server_key_iv: "7cdd76683ae6997875116ebaf23bd40a"
server_key_version: "encryptionkey@0"
I20260926 16:48:42.593947 27975 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 16:48:42.598228 27989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:42.600379 27975 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 16:48:42.600608 27975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "5e696c7a6fe3480da0c4091d5eebd052"
format_stamp: "Formatted at 2026-09-26 16:48:42 on dist-test-slave-wt2j"
server_key: "72076bfed53cfe5c4b67d92abbf0ba06"
server_key_iv: "7cdd76683ae6997875116ebaf23bd40a"
server_key_version: "encryptionkey@0"
I20260926 16:48:42.600863 27975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:48:42.621033 27975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:48:42.621923 27975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:48:42.622257 27975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:48:42.623385 27975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:48:42.625213 27975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:48:42.625372 27975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:42.625514 27975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:48:42.625608 27975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:42.665575 27975 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:37721
I20260926 16:48:42.665681 28101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:37721 every 8 connection(s)
I20260926 16:48:42.667034 27975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:48:42.668164 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 27975
I20260926 16:48:42.668391 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance
I20260926 16:48:42.669137 10171 external_mini_cluster.cc:1428] Setting key 582d41d4ff16d476614df30091da902c
I20260926 16:48:42.673604 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:0
--local_ip_for_outbound_sockets=127.9.238.194
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:34623
--builtin_ntp_servers=127.9.238.212:38223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 16:48:42.687659 28104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:42.670742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:33439 (local address 127.9.238.254:34623)
0926 16:48:42.671353 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:42.671360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:42.673172 (+ 1812us) server_negotiation.cc:407] Connection header received
0926 16:48:42.677021 (+ 3849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:42.677026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:42.677120 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:42.677245 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:42.679328 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:42.680176 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:42.683445 (+ 3269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:42.683987 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:42.684161 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:42.686423 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:42.686432 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:42.686440 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:48:42.686472 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:42.686534 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:48:42.686556 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:42.686671 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:42.687077 (+ 406us) server_negotiation.cc:299] Negotiation successful
0926 16:48:42.687204 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":276,"threads_started":1}
I20260926 16:48:42.689112 28105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:42.671511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:34623 (local address 127.9.238.193:33439)
0926 16:48:42.672949 (+ 1438us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:42.673010 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:42.676560 (+ 3550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:42.677656 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:42.677674 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:42.677847 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:48:42.679009 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:42.679048 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:42.680395 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:42.680405 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:42.683216 (+ 2811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:42.683241 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:42.684356 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:42.684365 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:42.684798 (+ 433us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:42.685863 (+ 1065us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:48:42.685869 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:48:42.685945 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:48:42.686104 (+ 159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:48:42.686722 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:48:42.686733 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:48:42.686742 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 16:48:42.686969 (+ 227us) client_negotiation.cc:240] Negotiation successful
0926 16:48:42.687445 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":371,"threads_started":1}
I20260926 16:48:42.691478 28102 heartbeater.cc:347] Connected to a master server at 127.9.238.254:34623
I20260926 16:48:42.691959 28102 heartbeater.cc:464] Registering TS with master...
I20260926 16:48:42.693071 28102 heartbeater.cc:511] Master 127.9.238.254:34623 requested a full tablet report, sending...
I20260926 16:48:42.695969 27916 ts_manager.cc:194] Registered new tserver with Master: 5e696c7a6fe3480da0c4091d5eebd052 (127.9.238.193:37721)
I20260926 16:48:42.697770 27916 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.238.193:33439
W20260926 16:48:43.021881 28106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:48:43.022205 28106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:48:43.022342 28106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:48:43.030048 28106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:48:43.030167 28106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:48:43.030339 28106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:48:43.040019 28106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:38223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:34623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:48:43.041424 28106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:48:43.043314 28106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:48:43.057747 28112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:43.057710 28111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:43.058346 28114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:48:43.059386 28106 server_base.cc:1034] running on GCE node
I20260926 16:48:43.060204 28106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:48:43.061394 28106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:48:43.062665 28106 hybrid_clock.cc:648] HybridClock initialized: now 1790441323062608 us; error 45 us; skew 500 ppm
I20260926 16:48:43.063136 28106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:48:43.065879 28106 webserver.cc:460] Webserver started at http://127.9.238.194:37399/ using document root <none> and password file <none>
I20260926 16:48:43.066640 28106 fs_manager.cc:359] Metadata directory not provided
I20260926 16:48:43.066731 28106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:48:43.067111 28106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:48:43.070339 28106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-1/data/instance:
uuid: "b040a31badc3441082665be0bf4cad87"
format_stamp: "Formatted at 2026-09-26 16:48:43 on dist-test-slave-wt2j"
server_key: "1ed07a7a6d71a8b0fe60e877284ef4f2"
server_key_iv: "a8ea8e982838bd8af5809c59f80f72bc"
server_key_version: "encryptionkey@0"
I20260926 16:48:43.071182 28106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance:
uuid: "b040a31badc3441082665be0bf4cad87"
format_stamp: "Formatted at 2026-09-26 16:48:43 on dist-test-slave-wt2j"
server_key: "1ed07a7a6d71a8b0fe60e877284ef4f2"
server_key_iv: "a8ea8e982838bd8af5809c59f80f72bc"
server_key_version: "encryptionkey@0"
I20260926 16:48:43.077127 28106 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 16:48:43.081429 28120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:43.083118 28106 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:48:43.083285 28106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "b040a31badc3441082665be0bf4cad87"
format_stamp: "Formatted at 2026-09-26 16:48:43 on dist-test-slave-wt2j"
server_key: "1ed07a7a6d71a8b0fe60e877284ef4f2"
server_key_iv: "a8ea8e982838bd8af5809c59f80f72bc"
server_key_version: "encryptionkey@0"
I20260926 16:48:43.083621 28106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:48:43.107656 28106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:48:43.108595 28106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:48:43.108925 28106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:48:43.110126 28106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:48:43.112032 28106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:48:43.112128 28106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:43.112304 28106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:48:43.112402 28106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:43.153108 28106 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:44463
I20260926 16:48:43.153276 28232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:44463 every 8 connection(s)
I20260926 16:48:43.154536 28106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:48:43.161563 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 28106
I20260926 16:48:43.161814 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance
I20260926 16:48:43.162339 10171 external_mini_cluster.cc:1428] Setting key 34fa5050475b829ad44ac25d0264ded8
I20260926 16:48:43.165952 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:0
--local_ip_for_outbound_sockets=127.9.238.195
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:34623
--builtin_ntp_servers=127.9.238.212:38223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 16:48:43.170555 28104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:43.158176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:39905 (local address 127.9.238.254:34623)
0926 16:48:43.158365 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:43.158370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:43.160488 (+ 2118us) server_negotiation.cc:407] Connection header received
0926 16:48:43.162819 (+ 2331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:43.162826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:43.162938 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:43.163066 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:43.164485 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.165343 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.167194 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.167746 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.167895 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:43.169667 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:43.169672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:43.169675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:48:43.169695 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:43.169727 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:48:43.169739 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:43.169838 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:43.170081 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 16:48:43.170186 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 16:48:43.171846 28235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:43.158780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:34623 (local address 127.9.238.194:39905)
0926 16:48:43.160256 (+ 1476us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:43.160309 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:43.162394 (+ 2085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:43.163336 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:43.163346 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:43.163509 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:48:43.164284 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:43.164304 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.165525 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.165531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:43.167018 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:43.167032 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.168016 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.168025 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:43.168296 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:43.169312 (+ 1016us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:48:43.169316 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:48:43.169359 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:48:43.169448 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:48:43.169835 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:48:43.169847 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:48:43.169858 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 16:48:43.170072 (+ 214us) client_negotiation.cc:240] Negotiation successful
0926 16:48:43.170551 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"thread_start_us":477,"threads_started":1}
I20260926 16:48:43.174023 28233 heartbeater.cc:347] Connected to a master server at 127.9.238.254:34623
I20260926 16:48:43.174445 28233 heartbeater.cc:464] Registering TS with master...
I20260926 16:48:43.175405 28233 heartbeater.cc:511] Master 127.9.238.254:34623 requested a full tablet report, sending...
I20260926 16:48:43.177579 27916 ts_manager.cc:194] Registered new tserver with Master: b040a31badc3441082665be0bf4cad87 (127.9.238.194:44463)
I20260926 16:48:43.178611 27916 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.238.194:39905
W20260926 16:48:43.510213 28236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:48:43.510571 28236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:48:43.510746 28236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:48:43.520407 28236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:48:43.520534 28236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:48:43.520714 28236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:48:43.530236 28236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:38223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:34623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:48:43.531715 28236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:48:43.533545 28236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:48:43.550017 28242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:43.550163 28241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:43.549981 28244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:48:43.551244 28236 server_base.cc:1034] running on GCE node
I20260926 16:48:43.552142 28236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:48:43.553416 28236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:48:43.554723 28236 hybrid_clock.cc:648] HybridClock initialized: now 1790441323554651 us; error 57 us; skew 500 ppm
I20260926 16:48:43.555194 28236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:48:43.567685 28236 webserver.cc:460] Webserver started at http://127.9.238.195:42995/ using document root <none> and password file <none>
I20260926 16:48:43.568850 28236 fs_manager.cc:359] Metadata directory not provided
I20260926 16:48:43.568957 28236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:48:43.569381 28236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:48:43.573328 28236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-2/data/instance:
uuid: "f07c050990fc4a039e91f8721d1e7570"
format_stamp: "Formatted at 2026-09-26 16:48:43 on dist-test-slave-wt2j"
server_key: "113a389c9bf3a7a816069556804abca8"
server_key_iv: "121683e3cae9fc915cc90a92f4253d09"
server_key_version: "encryptionkey@0"
I20260926 16:48:43.574359 28236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance:
uuid: "f07c050990fc4a039e91f8721d1e7570"
format_stamp: "Formatted at 2026-09-26 16:48:43 on dist-test-slave-wt2j"
server_key: "113a389c9bf3a7a816069556804abca8"
server_key_iv: "121683e3cae9fc915cc90a92f4253d09"
server_key_version: "encryptionkey@0"
I20260926 16:48:43.581409 28236 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260926 16:48:43.586575 28250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:43.588904 28236 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 16:48:43.589115 28236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "f07c050990fc4a039e91f8721d1e7570"
format_stamp: "Formatted at 2026-09-26 16:48:43 on dist-test-slave-wt2j"
server_key: "113a389c9bf3a7a816069556804abca8"
server_key_iv: "121683e3cae9fc915cc90a92f4253d09"
server_key_version: "encryptionkey@0"
I20260926 16:48:43.589368 28236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:48:43.604900 28236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:48:43.605937 28236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:48:43.606230 28236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:48:43.607371 28236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:48:43.609408 28236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:48:43.609534 28236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:43.609661 28236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:48:43.609724 28236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:43.651837 28236 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:40165
I20260926 16:48:43.651852 28362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:40165 every 8 connection(s)
I20260926 16:48:43.653486 28236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:48:43.660779 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 28236
I20260926 16:48:43.661010 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance
I20260926 16:48:43.661648 10171 external_mini_cluster.cc:1428] Setting key 3b1012b6b1d98d823c2cbf7caa609682
I20260926 16:48:43.670746 28104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:43.657230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:54755 (local address 127.9.238.254:34623)
0926 16:48:43.657676 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:43.657682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:43.659475 (+ 1793us) server_negotiation.cc:407] Connection header received
0926 16:48:43.662521 (+ 3046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:43.662529 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:43.662652 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:43.662754 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:43.664698 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.665788 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.667711 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.668131 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.668271 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:43.669795 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:43.669800 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:43.669803 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:48:43.669823 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:43.669853 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:48:43.669865 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:43.669964 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:43.670226 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 16:48:43.670368 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279}
I20260926 16:48:43.671903 28365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:43.657844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:34623 (local address 127.9.238.195:54755)
0926 16:48:43.659176 (+ 1332us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:43.659264 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:43.662056 (+ 2792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:43.663132 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:43.663142 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:43.663314 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:48:43.664452 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:43.664478 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.666066 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.666072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:43.667475 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:43.667494 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.668375 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.668383 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:43.668636 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:43.669430 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:48:43.669433 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:48:43.669477 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:48:43.669569 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:48:43.669972 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:48:43.669985 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:48:43.670004 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 16:48:43.670194 (+ 190us) client_negotiation.cc:240] Negotiation successful
0926 16:48:43.670625 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":371,"threads_started":1}
I20260926 16:48:43.673794 28363 heartbeater.cc:347] Connected to a master server at 127.9.238.254:34623
I20260926 16:48:43.674345 28363 heartbeater.cc:464] Registering TS with master...
I20260926 16:48:43.675382 28363 heartbeater.cc:511] Master 127.9.238.254:34623 requested a full tablet report, sending...
I20260926 16:48:43.677497 27915 ts_manager.cc:194] Registered new tserver with Master: f07c050990fc4a039e91f8721d1e7570 (127.9.238.195:40165)
I20260926 16:48:43.678633 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:48:43.678865 27915 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.238.195:54755
I20260926 16:48:43.690395 28104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:43.683759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33274 (local address 127.9.238.254:34623)
0926 16:48:43.684029 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:43.684038 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:43.684545 (+ 507us) server_negotiation.cc:407] Connection header received
0926 16:48:43.684772 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:43.684778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:43.684859 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:43.684968 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:43.685519 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.686250 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.687805 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.688344 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.688492 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:43.689676 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:43.689681 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:43.689684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:48:43.689703 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:43.689734 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:48:43.689746 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:43.689846 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:43.689985 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 16:48:43.690077 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 16:48:43.696069 27915 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 16:48:43.702057 28102 heartbeater.cc:503] Master 127.9.238.254:34623 was elected leader, sending a full tablet report...
W20260926 16:48:43.703375 27915 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:48:43.743661 28375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:43.720007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:44463 (local address 127.0.0.1:41194)
0926 16:48:43.726779 (+ 6772us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:43.726858 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:43.727139 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:43.734765 (+ 7626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:43.734777 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:43.734824 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:48:43.735656 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:43.735673 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.737818 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.737825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:43.740351 (+ 2526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:43.740365 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.742498 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.742506 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:43.742733 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:43.742789 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 16:48:43.742916 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 16:48:43.743146 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6323,"mutex_wait_us":25,"spinlock_wait_cycles":6272,"thread_start_us":6123,"threads_started":1}
I20260926 16:48:43.744160 28376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:43.720338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41194 (local address 127.9.238.194:44463)
0926 16:48:43.733556 (+ 13218us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:43.733564 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:43.733598 (+ 34us) server_negotiation.cc:407] Connection header received
0926 16:48:43.733703 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:43.733710 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:43.734116 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:43.734395 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:48:43.735976 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.737587 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.740587 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.741973 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.742107 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:43.742364 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:43.743423 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0926 16:48:43.743669 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13041,"thread_start_us":12852,"threads_started":1}
I20260926 16:48:43.744716 28378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:43.721416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49390 (local address 127.9.238.195:40165)
0926 16:48:43.733234 (+ 11818us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:43.733248 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:43.733911 (+ 663us) server_negotiation.cc:407] Connection header received
0926 16:48:43.734116 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:43.734123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:43.734343 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:43.734530 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:48:43.736638 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.738283 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.740892 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.742594 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.742721 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:43.743043 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:43.743764 (+ 721us) server_negotiation.cc:299] Negotiation successful
0926 16:48:43.744245 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11528,"thread_start_us":11290,"threads_started":1}
I20260926 16:48:43.744870 28377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:43.720613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:40165 (local address 127.0.0.1:49390)
0926 16:48:43.733749 (+ 13136us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:43.733770 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:43.733931 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:43.734831 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:43.734835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:43.734852 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:48:43.736391 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:43.736405 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.738437 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.738444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:43.740627 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:43.740643 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.743245 (+ 2602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.743262 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:43.743482 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:43.743517 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 16:48:43.744421 (+ 904us) client_negotiation.cc:240] Negotiation successful
0926 16:48:43.744592 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13007,"mutex_wait_us":704,"thread_start_us":12856,"threads_started":1}
I20260926 16:48:43.745891 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:43.734218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:37721 (local address 127.0.0.1:59090)
0926 16:48:43.735795 (+ 1577us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:43.735817 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:43.735957 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:43.737335 (+ 1378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:43.737339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:43.737364 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:48:43.737772 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:43.737785 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.739838 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.739844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:43.742076 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:43.742091 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.745281 (+ 3190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.745286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:43.745420 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:43.745453 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 16:48:43.745520 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 16:48:43.745615 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1405,"thread_start_us":1180,"threads_started":1}
I20260926 16:48:43.746522 28380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:43.734350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59090 (local address 127.9.238.193:37721)
0926 16:48:43.736365 (+ 2015us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:43.736377 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:43.736410 (+ 33us) server_negotiation.cc:407] Connection header received
0926 16:48:43.736533 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:43.736541 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:43.736789 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:43.737013 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:48:43.738006 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.739611 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.742339 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.743990 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.744129 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:43.745872 (+ 1743us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:43.745976 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 16:48:43.746189 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1763,"thread_start_us":1557,"threads_started":1}
I20260926 16:48:43.749128 28168 tablet_service.cc:1467] Processing CreateTablet for tablet cbc31152aa414110aa008e9252cd0191 (DEFAULT_TABLE table=test-table [id=d88b6eb1a09146358e19f974c44bf57d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:48:43.748998 28298 tablet_service.cc:1467] Processing CreateTablet for tablet cbc31152aa414110aa008e9252cd0191 (DEFAULT_TABLE table=test-table [id=d88b6eb1a09146358e19f974c44bf57d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:48:43.750584 28298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbc31152aa414110aa008e9252cd0191. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:48:43.750584 28168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbc31152aa414110aa008e9252cd0191. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:48:43.750371 28037 tablet_service.cc:1467] Processing CreateTablet for tablet cbc31152aa414110aa008e9252cd0191 (DEFAULT_TABLE table=test-table [id=d88b6eb1a09146358e19f974c44bf57d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:48:43.751775 28037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbc31152aa414110aa008e9252cd0191. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:48:43.768453 28381 tablet_bootstrap.cc:492] T cbc31152aa414110aa008e9252cd0191 P 5e696c7a6fe3480da0c4091d5eebd052: Bootstrap starting.
I20260926 16:48:43.769520 28382 tablet_bootstrap.cc:492] T cbc31152aa414110aa008e9252cd0191 P b040a31badc3441082665be0bf4cad87: Bootstrap starting.
I20260926 16:48:43.771164 28383 tablet_bootstrap.cc:492] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570: Bootstrap starting.
I20260926 16:48:43.774498 28381 tablet_bootstrap.cc:654] T cbc31152aa414110aa008e9252cd0191 P 5e696c7a6fe3480da0c4091d5eebd052: Neither blocks nor log segments found. Creating new log.
I20260926 16:48:43.775250 28383 tablet_bootstrap.cc:654] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570: Neither blocks nor log segments found. Creating new log.
I20260926 16:48:43.775249 28382 tablet_bootstrap.cc:654] T cbc31152aa414110aa008e9252cd0191 P b040a31badc3441082665be0bf4cad87: Neither blocks nor log segments found. Creating new log.
I20260926 16:48:43.776029 28381 log.cc:824] T cbc31152aa414110aa008e9252cd0191 P 5e696c7a6fe3480da0c4091d5eebd052: Log is configured to *not* fsync() on all Append() calls
I20260926 16:48:43.776765 28382 log.cc:824] T cbc31152aa414110aa008e9252cd0191 P b040a31badc3441082665be0bf4cad87: Log is configured to *not* fsync() on all Append() calls
I20260926 16:48:43.776762 28383 log.cc:824] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570: Log is configured to *not* fsync() on all Append() calls
I20260926 16:48:43.781358 28381 tablet_bootstrap.cc:492] T cbc31152aa414110aa008e9252cd0191 P 5e696c7a6fe3480da0c4091d5eebd052: No bootstrap required, opened a new log
I20260926 16:48:43.781368 28382 tablet_bootstrap.cc:492] T cbc31152aa414110aa008e9252cd0191 P b040a31badc3441082665be0bf4cad87: No bootstrap required, opened a new log
I20260926 16:48:43.781745 28382 ts_tablet_manager.cc:1397] T cbc31152aa414110aa008e9252cd0191 P b040a31badc3441082665be0bf4cad87: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.001s
I20260926 16:48:43.781745 28381 ts_tablet_manager.cc:1397] T cbc31152aa414110aa008e9252cd0191 P 5e696c7a6fe3480da0c4091d5eebd052: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.004s
I20260926 16:48:43.782521 28383 tablet_bootstrap.cc:492] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570: No bootstrap required, opened a new log
I20260926 16:48:43.782887 28383 ts_tablet_manager.cc:1397] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.008s
I20260926 16:48:43.792176 28383 raft_consensus.cc:348] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b040a31badc3441082665be0bf4cad87" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44463 } } peers { permanent_uuid: "f07c050990fc4a039e91f8721d1e7570" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40165 } } peers { permanent_uuid: "5e696c7a6fe3480da0c4091d5eebd052" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 37721 } }
I20260926 16:48:43.792795 28383 raft_consensus.cc:374] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:48:43.792901 28383 raft_consensus.cc:729] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f07c050990fc4a039e91f8721d1e7570, State: Initialized, Role: FOLLOWER
I20260926 16:48:43.793586 28383 consensus_queue.cc:260] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b040a31badc3441082665be0bf4cad87" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44463 } } peers { permanent_uuid: "f07c050990fc4a039e91f8721d1e7570" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40165 } } peers { permanent_uuid: "5e696c7a6fe3480da0c4091d5eebd052" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 37721 } }
I20260926 16:48:43.793785 28381 raft_consensus.cc:348] T cbc31152aa414110aa008e9252cd0191 P 5e696c7a6fe3480da0c4091d5eebd052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b040a31badc3441082665be0bf4cad87" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44463 } } peers { permanent_uuid: "f07c050990fc4a039e91f8721d1e7570" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40165 } } peers { permanent_uuid: "5e696c7a6fe3480da0c4091d5eebd052" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 37721 } }
I20260926 16:48:43.793773 28382 raft_consensus.cc:348] T cbc31152aa414110aa008e9252cd0191 P b040a31badc3441082665be0bf4cad87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b040a31badc3441082665be0bf4cad87" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44463 } } peers { permanent_uuid: "f07c050990fc4a039e91f8721d1e7570" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40165 } } peers { permanent_uuid: "5e696c7a6fe3480da0c4091d5eebd052" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 37721 } }
I20260926 16:48:43.794337 28381 raft_consensus.cc:374] T cbc31152aa414110aa008e9252cd0191 P 5e696c7a6fe3480da0c4091d5eebd052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:48:43.794451 28381 raft_consensus.cc:729] T cbc31152aa414110aa008e9252cd0191 P 5e696c7a6fe3480da0c4091d5eebd052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e696c7a6fe3480da0c4091d5eebd052, State: Initialized, Role: FOLLOWER
I20260926 16:48:43.794618 28382 raft_consensus.cc:374] T cbc31152aa414110aa008e9252cd0191 P b040a31badc3441082665be0bf4cad87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:48:43.794723 28382 raft_consensus.cc:729] T cbc31152aa414110aa008e9252cd0191 P b040a31badc3441082665be0bf4cad87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b040a31badc3441082665be0bf4cad87, State: Initialized, Role: FOLLOWER
I20260926 16:48:43.795579 28363 heartbeater.cc:503] Master 127.9.238.254:34623 was elected leader, sending a full tablet report...
I20260926 16:48:43.795447 28382 consensus_queue.cc:260] T cbc31152aa414110aa008e9252cd0191 P b040a31badc3441082665be0bf4cad87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b040a31badc3441082665be0bf4cad87" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44463 } } peers { permanent_uuid: "f07c050990fc4a039e91f8721d1e7570" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40165 } } peers { permanent_uuid: "5e696c7a6fe3480da0c4091d5eebd052" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 37721 } }
I20260926 16:48:43.795977 28383 ts_tablet_manager.cc:1428] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260926 16:48:43.796860 28233 heartbeater.cc:503] Master 127.9.238.254:34623 was elected leader, sending a full tablet report...
I20260926 16:48:43.797408 28381 consensus_queue.cc:260] T cbc31152aa414110aa008e9252cd0191 P 5e696c7a6fe3480da0c4091d5eebd052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b040a31badc3441082665be0bf4cad87" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44463 } } peers { permanent_uuid: "f07c050990fc4a039e91f8721d1e7570" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40165 } } peers { permanent_uuid: "5e696c7a6fe3480da0c4091d5eebd052" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 37721 } }
I20260926 16:48:43.797677 28382 ts_tablet_manager.cc:1428] T cbc31152aa414110aa008e9252cd0191 P b040a31badc3441082665be0bf4cad87: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
I20260926 16:48:43.800385 28381 ts_tablet_manager.cc:1428] T cbc31152aa414110aa008e9252cd0191 P 5e696c7a6fe3480da0c4091d5eebd052: Time spent starting tablet: real 0.019s user 0.008s sys 0.010s
I20260926 16:48:43.828287 28387 raft_consensus.cc:482] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:48:43.828747 28387 raft_consensus.cc:504] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b040a31badc3441082665be0bf4cad87" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44463 } } peers { permanent_uuid: "f07c050990fc4a039e91f8721d1e7570" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40165 } } peers { permanent_uuid: "5e696c7a6fe3480da0c4091d5eebd052" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 37721 } }
I20260926 16:48:43.831092 28387 leader_election.cc:288] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b040a31badc3441082665be0bf4cad87 (127.9.238.194:44463), 5e696c7a6fe3480da0c4091d5eebd052 (127.9.238.193:37721)
I20260926 16:48:43.838591 28376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:43.831525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:55325 (local address 127.9.238.194:44463)
0926 16:48:43.831801 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:43.831807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:43.831831 (+ 24us) server_negotiation.cc:407] Connection header received
0926 16:48:43.832080 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:43.832090 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:43.832218 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:43.832348 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:48:43.833101 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.834069 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.836654 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.837589 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.837773 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:43.837845 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:43.838138 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 16:48:43.838282 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260926 16:48:43.838587 28365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:43.831351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:44463 (local address 127.9.238.195:55325)
0926 16:48:43.831671 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:43.831692 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:43.831885 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:43.832415 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:43.832426 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:43.832467 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:48:43.832908 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:43.832927 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.834210 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.834214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:43.836468 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:43.836483 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.837766 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.837770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:43.837953 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:43.837985 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 16:48:43.838076 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 16:48:43.838200 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":4}
I20260926 16:48:43.839603 28380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:43.831955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:47509 (local address 127.9.238.193:37721)
0926 16:48:43.832208 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:43.832218 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:43.832985 (+ 767us) server_negotiation.cc:407] Connection header received
0926 16:48:43.833203 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:43.833210 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:43.833318 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:43.833427 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:48:43.833990 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.834728 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.837321 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.838463 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.838639 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:43.838728 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:43.839177 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 16:48:43.839321 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:48:43.839653 28390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:43.831794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:37721 (local address 127.9.238.195:47509)
0926 16:48:43.832840 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:43.832860 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:43.833011 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:43.833458 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:43.833462 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:43.833483 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:48:43.833820 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:43.833827 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.834902 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.834908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:43.837113 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:43.837124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.838678 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.838684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:43.838873 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:43.838911 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 16:48:43.839090 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 16:48:43.839234 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"mutex_wait_us":56,"thread_start_us":419,"threads_started":1}
I20260926 16:48:43.840240 28188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc31152aa414110aa008e9252cd0191" candidate_uuid: "f07c050990fc4a039e91f8721d1e7570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b040a31badc3441082665be0bf4cad87" is_pre_election: true
I20260926 16:48:43.840685 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc31152aa414110aa008e9252cd0191" candidate_uuid: "f07c050990fc4a039e91f8721d1e7570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e696c7a6fe3480da0c4091d5eebd052" is_pre_election: true
I20260926 16:48:43.840927 28188 raft_consensus.cc:2454] T cbc31152aa414110aa008e9252cd0191 P b040a31badc3441082665be0bf4cad87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f07c050990fc4a039e91f8721d1e7570 in term 0.
I20260926 16:48:43.841269 28057 raft_consensus.cc:2454] T cbc31152aa414110aa008e9252cd0191 P 5e696c7a6fe3480da0c4091d5eebd052 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f07c050990fc4a039e91f8721d1e7570 in term 0.
I20260926 16:48:43.841882 28253 leader_election.cc:302] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b040a31badc3441082665be0bf4cad87, f07c050990fc4a039e91f8721d1e7570; no voters:
I20260926 16:48:43.842382 28387 raft_consensus.cc:2781] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:48:43.842555 28387 raft_consensus.cc:482] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:48:43.842700 28387 raft_consensus.cc:3037] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:48:43.845445 28387 raft_consensus.cc:504] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b040a31badc3441082665be0bf4cad87" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44463 } } peers { permanent_uuid: "f07c050990fc4a039e91f8721d1e7570" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40165 } } peers { permanent_uuid: "5e696c7a6fe3480da0c4091d5eebd052" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 37721 } }
I20260926 16:48:43.846269 28387 leader_election.cc:288] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570 [CANDIDATE]: Term 1 election: Requested vote from peers b040a31badc3441082665be0bf4cad87 (127.9.238.194:44463), 5e696c7a6fe3480da0c4091d5eebd052 (127.9.238.193:37721)
I20260926 16:48:43.846975 28188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc31152aa414110aa008e9252cd0191" candidate_uuid: "f07c050990fc4a039e91f8721d1e7570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b040a31badc3441082665be0bf4cad87"
I20260926 16:48:43.847275 28188 raft_consensus.cc:3037] T cbc31152aa414110aa008e9252cd0191 P b040a31badc3441082665be0bf4cad87 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:48:43.847254 28057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbc31152aa414110aa008e9252cd0191" candidate_uuid: "f07c050990fc4a039e91f8721d1e7570" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e696c7a6fe3480da0c4091d5eebd052"
I20260926 16:48:43.847525 28057 raft_consensus.cc:3037] T cbc31152aa414110aa008e9252cd0191 P 5e696c7a6fe3480da0c4091d5eebd052 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:48:43.850078 28057 raft_consensus.cc:2454] T cbc31152aa414110aa008e9252cd0191 P 5e696c7a6fe3480da0c4091d5eebd052 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f07c050990fc4a039e91f8721d1e7570 in term 1.
I20260926 16:48:43.850152 28188 raft_consensus.cc:2454] T cbc31152aa414110aa008e9252cd0191 P b040a31badc3441082665be0bf4cad87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f07c050990fc4a039e91f8721d1e7570 in term 1.
I20260926 16:48:43.850653 28252 leader_election.cc:302] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e696c7a6fe3480da0c4091d5eebd052, f07c050990fc4a039e91f8721d1e7570; no voters:
I20260926 16:48:43.851209 28387 raft_consensus.cc:2781] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:48:43.851907 28387 raft_consensus.cc:686] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570 [term 1 LEADER]: Becoming Leader. State: Replica: f07c050990fc4a039e91f8721d1e7570, State: Running, Role: LEADER
I20260926 16:48:43.852502 28387 consensus_queue.cc:237] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b040a31badc3441082665be0bf4cad87" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44463 } } peers { permanent_uuid: "f07c050990fc4a039e91f8721d1e7570" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40165 } } peers { permanent_uuid: "5e696c7a6fe3480da0c4091d5eebd052" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 37721 } }
I20260926 16:48:43.858688 27914 catalog_manager.cc:5495] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570 reported cstate change: term changed from 0 to 1, leader changed from <none> to f07c050990fc4a039e91f8721d1e7570 (127.9.238.195). New cstate: current_term: 1 leader_uuid: "f07c050990fc4a039e91f8721d1e7570" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b040a31badc3441082665be0bf4cad87" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f07c050990fc4a039e91f8721d1e7570" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e696c7a6fe3480da0c4091d5eebd052" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 37721 } health_report { overall_health: UNKNOWN } } }
I20260926 16:48:43.898921 28378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:43.892689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49394 (local address 127.9.238.195:40165)
0926 16:48:43.892927 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:43.892933 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:43.892954 (+ 21us) server_negotiation.cc:407] Connection header received
0926 16:48:43.893097 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:43.893102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:43.893206 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:43.893438 (+ 232us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:48:43.893818 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.894938 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.896430 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.896928 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.897052 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:43.897637 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:48:43.898327 (+ 690us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:48:43.898475 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:43.898595 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 16:48:43.898680 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260926 16:48:43.905385 28364 tablet.cc:2369] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:48:43.906922 28234 tablet.cc:2369] T cbc31152aa414110aa008e9252cd0191 P b040a31badc3441082665be0bf4cad87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:48:43.909870 28188 raft_consensus.cc:1261] T cbc31152aa414110aa008e9252cd0191 P b040a31badc3441082665be0bf4cad87 [term 1 FOLLOWER]: Refusing update from remote peer f07c050990fc4a039e91f8721d1e7570: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:48:43.909857 28057 raft_consensus.cc:1261] T cbc31152aa414110aa008e9252cd0191 P 5e696c7a6fe3480da0c4091d5eebd052 [term 1 FOLLOWER]: Refusing update from remote peer f07c050990fc4a039e91f8721d1e7570: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:48:43.911216 28391 consensus_queue.cc:1035] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b040a31badc3441082665be0bf4cad87" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:48:43.911651 28387 consensus_queue.cc:1035] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e696c7a6fe3480da0c4091d5eebd052" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 16:48:43.920034 28103 tablet.cc:2369] T cbc31152aa414110aa008e9252cd0191 P 5e696c7a6fe3480da0c4091d5eebd052: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:48:43.923902 28394 mvcc.cc:205] Tried to move back new op lower bound from 7333647662719774720 to 7333647662505385984. Current Snapshot: MvccSnapshot[applied={T|T < 7333647662719774720}]
I20260926 16:48:43.928191 28397 mvcc.cc:205] Tried to move back new op lower bound from 7333647662719774720 to 7333647662505385984. Current Snapshot: MvccSnapshot[applied={T|T < 7333647662719774720}]
I20260926 16:48:43.928653 28395 mvcc.cc:205] Tried to move back new op lower bound from 7333647662719774720 to 7333647662505385984. Current Snapshot: MvccSnapshot[applied={T|T < 7333647662719774720}]
I20260926 16:48:43.943061 27914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:48:43.943391 27914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:48:43.948144 27914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c7999080535453bb1933073ef8fbc58: Sending DeleteTablet for 3 replicas of tablet cbc31152aa414110aa008e9252cd0191
I20260926 16:48:43.949751 28168 tablet_service.cc:1514] Processing DeleteTablet for tablet cbc31152aa414110aa008e9252cd0191 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:48:43 UTC) from {username='slave'} at 127.0.0.1:41194
I20260926 16:48:43.949841 28037 tablet_service.cc:1514] Processing DeleteTablet for tablet cbc31152aa414110aa008e9252cd0191 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:48:43 UTC) from {username='slave'} at 127.0.0.1:59090
I20260926 16:48:43.950830 28298 tablet_service.cc:1514] Processing DeleteTablet for tablet cbc31152aa414110aa008e9252cd0191 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:48:43 UTC) from {username='slave'} at 127.0.0.1:49390
I20260926 16:48:43.952450 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 27975
I20260926 16:48:43.962522 28406 tablet_replica.cc:330] T cbc31152aa414110aa008e9252cd0191 P b040a31badc3441082665be0bf4cad87: stopping tablet replica
I20260926 16:48:43.963492 28406 raft_consensus.cc:2229] T cbc31152aa414110aa008e9252cd0191 P b040a31badc3441082665be0bf4cad87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:48:43.964461 28406 raft_consensus.cc:2258] T cbc31152aa414110aa008e9252cd0191 P b040a31badc3441082665be0bf4cad87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:48:43.965229 28405 tablet_replica.cc:330] T cbc31152aa414110aa008e9252cd0191 P 5e696c7a6fe3480da0c4091d5eebd052: stopping tablet replica
I20260926 16:48:43.965797 28405 raft_consensus.cc:2229] T cbc31152aa414110aa008e9252cd0191 P 5e696c7a6fe3480da0c4091d5eebd052 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:48:43.966387 28405 raft_consensus.cc:2258] T cbc31152aa414110aa008e9252cd0191 P 5e696c7a6fe3480da0c4091d5eebd052 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:48:43.968479 28406 ts_tablet_manager.cc:1905] T cbc31152aa414110aa008e9252cd0191 P b040a31badc3441082665be0bf4cad87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:48:43.973565 28407 tablet_replica.cc:330] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570: stopping tablet replica
I20260926 16:48:43.974094 28407 raft_consensus.cc:2229] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:48:43.974967 28407 raft_consensus.cc:2258] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:48:43.975072 28405 ts_tablet_manager.cc:1905] T cbc31152aa414110aa008e9252cd0191 P 5e696c7a6fe3480da0c4091d5eebd052: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:48:43.979377 28406 ts_tablet_manager.cc:1918] T cbc31152aa414110aa008e9252cd0191 P b040a31badc3441082665be0bf4cad87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:48:43.979710 28406 log.cc:1196] T cbc31152aa414110aa008e9252cd0191 P b040a31badc3441082665be0bf4cad87: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-1/wal/wals/cbc31152aa414110aa008e9252cd0191
I20260926 16:48:43.980047 28380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:43.954713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59092 (local address 127.9.238.193:37721)
0926 16:48:43.955009 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:43.955020 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:43.968595 (+ 13575us) server_negotiation.cc:407] Connection header received
0926 16:48:43.968932 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:43.968938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:43.969048 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:43.969132 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:43.969800 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.970942 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.975434 (+ 4492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:43.976245 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:43.976365 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:43.978023 (+ 1658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:43.978035 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:43.978054 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:48:43.978906 (+ 852us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:43.978992 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:48:43.979020 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:43.979111 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:43.979372 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 16:48:43.979521 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 16:48:43.980825 28406 ts_tablet_manager.cc:1939] T cbc31152aa414110aa008e9252cd0191 P b040a31badc3441082665be0bf4cad87: Deleting consensus metadata
I20260926 16:48:43.981441 28097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:48:43.982756 28407 ts_tablet_manager.cc:1905] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:48:43.986591 27902 catalog_manager.cc:4841] TS b040a31badc3441082665be0bf4cad87 (127.9.238.194:44463): tablet cbc31152aa414110aa008e9252cd0191 (table test-table [id=d88b6eb1a09146358e19f974c44bf57d]) successfully deleted
I20260926 16:48:43.992184 28407 ts_tablet_manager.cc:1918] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:48:43.992406 28407 log.cc:1196] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-2/wal/wals/cbc31152aa414110aa008e9252cd0191
I20260926 16:48:43.993037 28407 ts_tablet_manager.cc:1939] T cbc31152aa414110aa008e9252cd0191 P f07c050990fc4a039e91f8721d1e7570: Deleting consensus metadata
I20260926 16:48:43.995463 27901 catalog_manager.cc:4841] TS f07c050990fc4a039e91f8721d1e7570 (127.9.238.195:40165): tablet cbc31152aa414110aa008e9252cd0191 (table test-table [id=d88b6eb1a09146358e19f974c44bf57d]) successfully deleted
I20260926 16:48:44.100402 28405 ts_tablet_manager.cc:1918] T cbc31152aa414110aa008e9252cd0191 P 5e696c7a6fe3480da0c4091d5eebd052: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:48:44.100756 28405 log.cc:1196] T cbc31152aa414110aa008e9252cd0191 P 5e696c7a6fe3480da0c4091d5eebd052: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790440988577176-10171-0/minicluster-data/ts-0/wal/wals/cbc31152aa414110aa008e9252cd0191
I20260926 16:48:44.101284 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 27975
I20260926 16:48:44.101442 28405 ts_tablet_manager.cc:1939] T cbc31152aa414110aa008e9252cd0191 P 5e696c7a6fe3480da0c4091d5eebd052: Deleting consensus metadata
I20260926 16:48:44.124321 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 28106
W20260926 16:48:44.124351 27901 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.238.193:37721 (error 108)
I20260926 16:48:44.126590 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:44.126074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:37721 (local address 127.0.0.1:59094)
0926 16:48:44.126299 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:44.126418 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 16:48:44.127071 27901 catalog_manager.cc:4584] TS 5e696c7a6fe3480da0c4091d5eebd052 (127.9.238.193:37721): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cbc31152aa414110aa008e9252cd0191: Network error: Client connection negotiation failed: client connection to 127.9.238.193:37721: connect: Connection refused (error 111)
I20260926 16:48:44.133558 28376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:44.126096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41198 (local address 127.9.238.194:44463)
0926 16:48:44.126298 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:44.126304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:44.126323 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:48:44.126391 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:44.126397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:44.126528 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:44.126651 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:44.127334 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:44.128385 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:44.129804 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:44.130259 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:44.130423 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:44.131889 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:44.131900 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:44.131920 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:48:44.132695 (+ 775us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:44.132757 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:48:44.132789 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:44.132951 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:44.133046 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 16:48:44.133177 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 16:48:44.134586 28228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:48:44.181187 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:44.180391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:37721 (local address 127.0.0.1:59102)
0926 16:48:44.180738 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:44.180912 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260926 16:48:44.182832 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:44.182246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:37721 (local address 127.0.0.1:59110)
0926 16:48:44.182475 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:44.182614 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 16:48:44.236366 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 28106
I20260926 16:48:44.243826 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:44.243158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:37721 (local address 127.0.0.1:59116)
0926 16:48:44.243397 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:44.243515 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 16:48:44.245052 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:44.244699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:37721 (local address 127.0.0.1:59118)
0926 16:48:44.244854 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:44.244937 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 16:48:44.262141 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 28236
I20260926 16:48:44.271811 28378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:44.263266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49400 (local address 127.9.238.195:40165)
0926 16:48:44.263513 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:44.263523 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:44.263603 (+ 80us) server_negotiation.cc:407] Connection header received
0926 16:48:44.263926 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:44.263932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:44.264048 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:44.264159 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:44.264748 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:44.265699 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:44.267934 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:44.268415 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:44.268531 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:44.269978 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:44.269988 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:44.270007 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:48:44.270828 (+ 821us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:44.270913 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:48:44.270938 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:44.271047 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:44.271225 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 16:48:44.271325 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 16:48:44.272809 28358 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:48:44.326356 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:44.325554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:37721 (local address 127.0.0.1:59126)
0926 16:48:44.325913 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:44.326076 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260926 16:48:44.327875 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:44.327321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:37721 (local address 127.0.0.1:59130)
0926 16:48:44.327616 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:44.327720 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260926 16:48:44.424511 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 28236
I20260926 16:48:44.447638 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 27885
I20260926 16:48:44.449268 27946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:48:44.567412 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 27885
I20260926 16:48:44.567589 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:44.566738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:37721 (local address 127.0.0.1:59140)
0926 16:48:44.567166 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:44.567310 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
2026-09-26T16:48:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3076 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:48:44 dist-test-slave-wt2j krb5kdc[28416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:48:44 dist-test-slave-wt2j krb5kdc[28416](info): set up 2 sockets
Sep 26 16:48:44 dist-test-slave-wt2j krb5kdc[28416](info): commencing operation
krb5kdc: starting...
W20260926 16:48:45.749374 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.002s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:48:45 dist-test-slave-wt2j krb5kdc[28416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:48:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:48:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:48:46.056200 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:46695
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:46695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:48:46.398031 28429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:48:46.398398 28429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:48:46.398572 28429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:48:46.406044 28429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:48:46.406222 28429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:48:46.406337 28429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:48:46.406435 28429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:48:46.406520 28429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:48:46.416869 28429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:46695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:46695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:48:46.418272 28429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:48:46.419626 28429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:48:46.432608 28434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:46.433153 28435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:46.433457 28437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:48:46.434082 28429 server_base.cc:1034] running on GCE node
I20260926 16:48:46.435166 28429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:48:46.437274 28429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:48:46.438547 28429 hybrid_clock.cc:648] HybridClock initialized: now 1790441326438495 us; error 50 us; skew 500 ppm
I20260926 16:48:46.439051 28429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:48:46 dist-test-slave-wt2j krb5kdc[28416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441326, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:48:46.443426 28429 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:48:46.445685 28429 webserver.cc:460] Webserver started at http://127.9.238.254:33251/ using document root <none> and password file <none>
I20260926 16:48:46.446761 28429 fs_manager.cc:359] Metadata directory not provided
I20260926 16:48:46.446942 28429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:48:46.447335 28429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:48:46.451237 28429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "9376ad89cd144f4d876330f8d435a3f7"
format_stamp: "Formatted at 2026-09-26 16:48:46 on dist-test-slave-wt2j"
server_key: "24bce835c9e81f14c482e8c847038164"
server_key_iv: "c359f25623194b1d6f67d96c9415f9cd"
server_key_version: "encryptionkey@0"
I20260926 16:48:46.452258 28429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "9376ad89cd144f4d876330f8d435a3f7"
format_stamp: "Formatted at 2026-09-26 16:48:46 on dist-test-slave-wt2j"
server_key: "24bce835c9e81f14c482e8c847038164"
server_key_iv: "c359f25623194b1d6f67d96c9415f9cd"
server_key_version: "encryptionkey@0"
I20260926 16:48:46.460196 28429 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260926 16:48:46.465485 28444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:46.467753 28429 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.007s sys 0.000s
I20260926 16:48:46.468070 28429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "9376ad89cd144f4d876330f8d435a3f7"
format_stamp: "Formatted at 2026-09-26 16:48:46 on dist-test-slave-wt2j"
server_key: "24bce835c9e81f14c482e8c847038164"
server_key_iv: "c359f25623194b1d6f67d96c9415f9cd"
server_key_version: "encryptionkey@0"
I20260926 16:48:46.468369 28429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:48:46.515851 28429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:48:46.522390 28429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:48:46.522773 28429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:48:46.547674 28429 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:46695
I20260926 16:48:46.547674 28495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:46695 every 8 connection(s)
I20260926 16:48:46.549562 28429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:48:46.553889 28496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:48:46.559044 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 28429
I20260926 16:48:46.559255 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:48:46.559860 10171 external_mini_cluster.cc:1428] Setting key 0e96c21fe3c2353eeea8c2e26d29ab4e
I20260926 16:48:46.565973 28496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9376ad89cd144f4d876330f8d435a3f7: Bootstrap starting.
I20260926 16:48:46.569531 28496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9376ad89cd144f4d876330f8d435a3f7: Neither blocks nor log segments found. Creating new log.
I20260926 16:48:46.570657 28496 log.cc:824] T 00000000000000000000000000000000 P 9376ad89cd144f4d876330f8d435a3f7: Log is configured to *not* fsync() on all Append() calls
I20260926 16:48:46.573792 28496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9376ad89cd144f4d876330f8d435a3f7: No bootstrap required, opened a new log
Sep 26 16:48:46 dist-test-slave-wt2j krb5kdc[28416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:48:46.582576 28496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9376ad89cd144f4d876330f8d435a3f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9376ad89cd144f4d876330f8d435a3f7" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46695 } }
I20260926 16:48:46.583125 28496 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9376ad89cd144f4d876330f8d435a3f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:48:46.583232 28496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9376ad89cd144f4d876330f8d435a3f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9376ad89cd144f4d876330f8d435a3f7, State: Initialized, Role: FOLLOWER
I20260926 16:48:46.584121 28496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9376ad89cd144f4d876330f8d435a3f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9376ad89cd144f4d876330f8d435a3f7" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46695 } }
I20260926 16:48:46.584645 28496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9376ad89cd144f4d876330f8d435a3f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:48:46.585172 28496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9376ad89cd144f4d876330f8d435a3f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:48:46.585533 28496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9376ad89cd144f4d876330f8d435a3f7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:48:46.589270 28496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9376ad89cd144f4d876330f8d435a3f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9376ad89cd144f4d876330f8d435a3f7" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46695 } }
I20260926 16:48:46.589965 28496 leader_election.cc:302] T 00000000000000000000000000000000 P 9376ad89cd144f4d876330f8d435a3f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9376ad89cd144f4d876330f8d435a3f7; no voters:
I20260926 16:48:46.590860 28496 leader_election.cc:288] T 00000000000000000000000000000000 P 9376ad89cd144f4d876330f8d435a3f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:48:46.591171 28501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9376ad89cd144f4d876330f8d435a3f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:48:46.594656 28501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9376ad89cd144f4d876330f8d435a3f7 [term 1 LEADER]: Becoming Leader. State: Replica: 9376ad89cd144f4d876330f8d435a3f7, State: Running, Role: LEADER
I20260926 16:48:46.595221 28496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9376ad89cd144f4d876330f8d435a3f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:48:46.595650 28501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9376ad89cd144f4d876330f8d435a3f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9376ad89cd144f4d876330f8d435a3f7" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46695 } }
I20260926 16:48:46.599416 28499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:46.562633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53518 (local address 127.9.238.254:46695)
0926 16:48:46.563836 (+ 1203us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:46.563851 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:46.563913 (+ 62us) server_negotiation.cc:407] Connection header received
0926 16:48:46.566094 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:46.566139 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:46.566747 (+ 608us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:46.567293 (+ 546us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:46.569188 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:46.570839 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:46.572568 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:46.573826 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:46.574087 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:46.581904 (+ 7817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:46.581923 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:46.581953 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:48:46.582028 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:46.589254 (+ 7226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:46.590789 (+ 1535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:46.590804 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:46.590818 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:46.591165 (+ 347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:46.592077 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:46.592087 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:46.592105 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:46.595140 (+ 3035us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:46.595711 (+ 571us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:46.596668 (+ 957us) server_negotiation.cc:299] Negotiation successful
0926 16:48:46.597336 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":917,"thread_start_us":583,"threads_started":1}
I20260926 16:48:46.599524 28502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9376ad89cd144f4d876330f8d435a3f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9376ad89cd144f4d876330f8d435a3f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9376ad89cd144f4d876330f8d435a3f7" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46695 } } }
I20260926 16:48:46.600103 28502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9376ad89cd144f4d876330f8d435a3f7 [sys.catalog]: This master's current role is: LEADER
I20260926 16:48:46.600787 28503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9376ad89cd144f4d876330f8d435a3f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9376ad89cd144f4d876330f8d435a3f7. Latest consensus state: current_term: 1 leader_uuid: "9376ad89cd144f4d876330f8d435a3f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9376ad89cd144f4d876330f8d435a3f7" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46695 } } }
I20260926 16:48:46.601169 28503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9376ad89cd144f4d876330f8d435a3f7 [sys.catalog]: This master's current role is: LEADER
I20260926 16:48:46.609355 28506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:48:46.615415 28506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:48:46.628037 28506 catalog_manager.cc:1347] Generated new cluster ID: fa7d81725b05460b977863ce1ee38a62
I20260926 16:48:46.628422 28506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:48:46.651602 28506 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:48:46.653573 28506 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:48:46.663347 28506 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9376ad89cd144f4d876330f8d435a3f7: Generated new TSK 0
I20260926 16:48:46.664943 28506 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
I20260926 16:48:46.752014 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:0
--local_ip_for_outbound_sockets=127.9.238.193
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:46695
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:48:47.181041 28522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:48:47.181409 28522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:48:47.181581 28522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:48:47.189759 28522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:48:47.189914 28522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:48:47.190112 28522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:48:47.201493 28522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:46695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:48:47.203277 28522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:48:47.205140 28522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:48:47.222165 28530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:47.222316 28528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:47.222306 28527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:48:47.223461 28522 server_base.cc:1034] running on GCE node
I20260926 16:48:47.224313 28522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:48:47.225637 28522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:48:47.226925 28522 hybrid_clock.cc:648] HybridClock initialized: now 1790441327226851 us; error 54 us; skew 500 ppm
I20260926 16:48:47.227414 28522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:48:47 dist-test-slave-wt2j krb5kdc[28416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441327, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:48:47.231386 28522 init.cc:377] Logged in from keytab as kudu/127.9.238.193@KRBTEST.COM (short username kudu)
I20260926 16:48:47.233294 28522 webserver.cc:460] Webserver started at http://127.9.238.193:42293/ using document root <none> and password file <none>
I20260926 16:48:47.234171 28522 fs_manager.cc:359] Metadata directory not provided
I20260926 16:48:47.234298 28522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:48:47.234611 28522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:48:47.238356 28522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-0/data/instance:
uuid: "7db29de44e1842ec80a015e99822f371"
format_stamp: "Formatted at 2026-09-26 16:48:47 on dist-test-slave-wt2j"
server_key: "fef0d80fa4b95480161ae93c467de80b"
server_key_iv: "3f1afe99e3859a8d542879b8a9f4054a"
server_key_version: "encryptionkey@0"
I20260926 16:48:47.239223 28522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance:
uuid: "7db29de44e1842ec80a015e99822f371"
format_stamp: "Formatted at 2026-09-26 16:48:47 on dist-test-slave-wt2j"
server_key: "fef0d80fa4b95480161ae93c467de80b"
server_key_iv: "3f1afe99e3859a8d542879b8a9f4054a"
server_key_version: "encryptionkey@0"
I20260926 16:48:47.245918 28522 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260926 16:48:47.250861 28537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:47.252948 28522 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 16:48:47.253196 28522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "7db29de44e1842ec80a015e99822f371"
format_stamp: "Formatted at 2026-09-26 16:48:47 on dist-test-slave-wt2j"
server_key: "fef0d80fa4b95480161ae93c467de80b"
server_key_iv: "3f1afe99e3859a8d542879b8a9f4054a"
server_key_version: "encryptionkey@0"
I20260926 16:48:47.253439 28522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:48:47.277697 28522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:48:47.282783 28522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:48:47.283118 28522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:48:47.284399 28522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:48:47.286034 28522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:48:47.286116 28522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:47.286250 28522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:48:47.286319 28522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:47.325925 28522 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:38691
I20260926 16:48:47.326028 28649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:38691 every 8 connection(s)
I20260926 16:48:47.327497 28522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:48:47.337436 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 28522
I20260926 16:48:47.337697 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance
I20260926 16:48:47.338171 10171 external_mini_cluster.cc:1428] Setting key d4daf2258e937eaa3c30c3166c57c221
Sep 26 16:48:47 dist-test-slave-wt2j krb5kdc[28416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441327, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:48:47.353674 28652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:47.331082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:53703 (local address 127.9.238.254:46695)
0926 16:48:47.331765 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:47.331771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:47.332750 (+ 979us) server_negotiation.cc:407] Connection header received
0926 16:48:47.335149 (+ 2399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:47.335156 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:47.335304 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:47.335459 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:47.337588 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:47.338595 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:47.340882 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:47.341401 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:47.341585 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:47.346606 (+ 5021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:47.346612 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:47.346616 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:48:47.346662 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:47.349695 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:47.350667 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:47.350672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:47.350674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:47.350835 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:47.351363 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:47.351369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:47.351387 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:47.351852 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:47.352025 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:47.352960 (+ 935us) server_negotiation.cc:299] Negotiation successful
0926 16:48:47.353253 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":364,"threads_started":1}
I20260926 16:48:47.354962 28653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:47.331501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46695 (local address 127.9.238.193:53703)
0926 16:48:47.332525 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:47.332586 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:47.334775 (+ 2189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:47.335817 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:47.335827 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:47.336590 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:48:47.337361 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:47.337384 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:47.338792 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:47.338796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:47.340686 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:47.340712 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:47.341729 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:47.341738 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:47.342041 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:47.343003 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:48:47.343033 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:48:47.346159 (+ 3126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:48:47.349919 (+ 3760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:48:47.349938 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:48:47.349959 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:48:47.350505 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:48:47.350985 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:48:47.350990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:48:47.350994 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:48:47.351225 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:48:47.352014 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:48:47.352024 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:48:47.352507 (+ 483us) client_negotiation.cc:769] Sending connection context
0926 16:48:47.352832 (+ 325us) client_negotiation.cc:240] Negotiation successful
0926 16:48:47.353547 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":339,"threads_started":1}
I20260926 16:48:47.357144 28650 heartbeater.cc:347] Connected to a master server at 127.9.238.254:46695
I20260926 16:48:47.357664 28650 heartbeater.cc:464] Registering TS with master...
I20260926 16:48:47.358687 28650 heartbeater.cc:511] Master 127.9.238.254:46695 requested a full tablet report, sending...
I20260926 16:48:47.361781 28461 ts_manager.cc:194] Registered new tserver with Master: 7db29de44e1842ec80a015e99822f371 (127.9.238.193:38691)
I20260926 16:48:47.364254 28461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.193@KRBTEST.COM'} at 127.9.238.193:53703
WARNING: no policy specified for kudu/127.9.238.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
I20260926 16:48:47.400506 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:0
--local_ip_for_outbound_sockets=127.9.238.194
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:46695
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:48:47.769731 28656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:48:47.770037 28656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:48:47.770220 28656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:48:47.777894 28656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:48:47.778028 28656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:48:47.778147 28656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:48:47.788370 28656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:46695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:48:47.789634 28656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:48:47.790937 28656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:48:47.807016 28661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:47.807067 28664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:47.807129 28662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:48:47.808456 28656 server_base.cc:1034] running on GCE node
I20260926 16:48:47.809216 28656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:48:47.810312 28656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:48:47.811618 28656 hybrid_clock.cc:648] HybridClock initialized: now 1790441327811484 us; error 72 us; skew 500 ppm
I20260926 16:48:47.812074 28656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:48:47 dist-test-slave-wt2j krb5kdc[28416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441327, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:48:47.816007 28656 init.cc:377] Logged in from keytab as kudu/127.9.238.194@KRBTEST.COM (short username kudu)
I20260926 16:48:47.817934 28656 webserver.cc:460] Webserver started at http://127.9.238.194:33241/ using document root <none> and password file <none>
I20260926 16:48:47.818754 28656 fs_manager.cc:359] Metadata directory not provided
I20260926 16:48:47.818905 28656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:48:47.819278 28656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:48:47.822438 28656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-1/data/instance:
uuid: "b8828e77745049dd936654ad51dc5a70"
format_stamp: "Formatted at 2026-09-26 16:48:47 on dist-test-slave-wt2j"
server_key: "4c6ed4af15a5111c2a0e3fed514ac0aa"
server_key_iv: "c628640a15b85b760cc62b2efff727bc"
server_key_version: "encryptionkey@0"
I20260926 16:48:47.823266 28656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance:
uuid: "b8828e77745049dd936654ad51dc5a70"
format_stamp: "Formatted at 2026-09-26 16:48:47 on dist-test-slave-wt2j"
server_key: "4c6ed4af15a5111c2a0e3fed514ac0aa"
server_key_iv: "c628640a15b85b760cc62b2efff727bc"
server_key_version: "encryptionkey@0"
I20260926 16:48:47.829638 28656 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260926 16:48:47.834616 28671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:47.836445 28656 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 16:48:47.836674 28656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "b8828e77745049dd936654ad51dc5a70"
format_stamp: "Formatted at 2026-09-26 16:48:47 on dist-test-slave-wt2j"
server_key: "4c6ed4af15a5111c2a0e3fed514ac0aa"
server_key_iv: "c628640a15b85b760cc62b2efff727bc"
server_key_version: "encryptionkey@0"
I20260926 16:48:47.836920 28656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:48:47.867985 28656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:48:47.873181 28656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:48:47.873610 28656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:48:47.874802 28656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:48:47.876470 28656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:48:47.876600 28656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:47.876773 28656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:48:47.876866 28656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:47.918069 28656 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:41071
I20260926 16:48:47.918177 28783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:41071 every 8 connection(s)
I20260926 16:48:47.919634 28656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:48:47.920161 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 28656
I20260926 16:48:47.920395 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance
I20260926 16:48:47.921116 10171 external_mini_cluster.cc:1428] Setting key 6644fe853f8f3b36002415c77b60ea80
Sep 26 16:48:47 dist-test-slave-wt2j krb5kdc[28416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441327, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:48:47.947301 28786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:47.923124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:42773 (local address 127.9.238.254:46695)
0926 16:48:47.923787 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:47.923794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:47.925552 (+ 1758us) server_negotiation.cc:407] Connection header received
0926 16:48:47.927929 (+ 2377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:47.927937 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:47.928046 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:47.928199 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:47.930261 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:47.931513 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:47.933466 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:47.933907 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:47.934115 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:47.939443 (+ 5328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:47.939449 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:47.939452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:48:47.939500 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:47.942848 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:47.944184 (+ 1336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:47.944188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:47.944191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:47.944358 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:47.944979 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:47.944982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:47.944984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:47.945304 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:47.945434 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:47.946695 (+ 1261us) server_negotiation.cc:299] Negotiation successful
0926 16:48:47.946923 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":306,"threads_started":1}
I20260926 16:48:47.949270 28787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:47.924202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46695 (local address 127.9.238.194:42773)
0926 16:48:47.925288 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:47.925353 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:47.927519 (+ 2166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:47.928536 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:47.928547 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:47.929263 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:48:47.930041 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:47.930063 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:47.931739 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:47.931744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:47.933207 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:47.933222 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:47.934257 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:47.934267 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:47.934641 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:47.935525 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:48:47.935581 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:48:47.939078 (+ 3497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:48:47.943073 (+ 3995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:48:47.943090 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:48:47.943118 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:48:47.944003 (+ 885us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:48:47.944498 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:48:47.944506 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:48:47.944512 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:48:47.944826 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:48:47.945451 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:48:47.945465 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:48:47.946146 (+ 681us) client_negotiation.cc:769] Sending connection context
0926 16:48:47.946553 (+ 407us) client_negotiation.cc:240] Negotiation successful
0926 16:48:47.947370 (+ 817us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":474,"threads_started":1}
I20260926 16:48:47.951331 28784 heartbeater.cc:347] Connected to a master server at 127.9.238.254:46695
I20260926 16:48:47.951834 28784 heartbeater.cc:464] Registering TS with master...
I20260926 16:48:47.952730 28784 heartbeater.cc:511] Master 127.9.238.254:46695 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.9.238.195@KRBTEST.COM; defaulting to no policy
I20260926 16:48:47.954459 28461 ts_manager.cc:194] Registered new tserver with Master: b8828e77745049dd936654ad51dc5a70 (127.9.238.194:41071)
I20260926 16:48:47.955411 28461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.194@KRBTEST.COM'} at 127.9.238.194:42773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
I20260926 16:48:47.993322 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:0
--local_ip_for_outbound_sockets=127.9.238.195
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:46695
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:46719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:48:48.368664 28650 heartbeater.cc:503] Master 127.9.238.254:46695 was elected leader, sending a full tablet report...
W20260926 16:48:48.424031 28790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:48:48.424428 28790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:48:48.424542 28790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:48:48.433072 28790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:48:48.433231 28790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:48:48.433367 28790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:48:48.445156 28790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:46719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:46695
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:48:48.446921 28790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:48:48.448698 28790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:48:48.466200 28795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:48.466337 28796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:48.466367 28798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:48:48.467401 28790 server_base.cc:1034] running on GCE node
I20260926 16:48:48.468370 28790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:48:48.469712 28790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:48:48.470988 28790 hybrid_clock.cc:648] HybridClock initialized: now 1790441328470921 us; error 48 us; skew 500 ppm
I20260926 16:48:48.471508 28790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:48:48 dist-test-slave-wt2j krb5kdc[28416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441328, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:48:48.475971 28790 init.cc:377] Logged in from keytab as kudu/127.9.238.195@KRBTEST.COM (short username kudu)
I20260926 16:48:48.477974 28790 webserver.cc:460] Webserver started at http://127.9.238.195:38747/ using document root <none> and password file <none>
I20260926 16:48:48.478893 28790 fs_manager.cc:359] Metadata directory not provided
I20260926 16:48:48.479060 28790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:48:48.479471 28790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:48:48.483299 28790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-2/data/instance:
uuid: "e5e43bb1ffd94cbd9ca290add2b4065b"
format_stamp: "Formatted at 2026-09-26 16:48:48 on dist-test-slave-wt2j"
server_key: "eeb0420774e0c4d8ded1fff24b55c373"
server_key_iv: "8d99185e84634ea6f68ea702ee516a2b"
server_key_version: "encryptionkey@0"
I20260926 16:48:48.484326 28790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance:
uuid: "e5e43bb1ffd94cbd9ca290add2b4065b"
format_stamp: "Formatted at 2026-09-26 16:48:48 on dist-test-slave-wt2j"
server_key: "eeb0420774e0c4d8ded1fff24b55c373"
server_key_iv: "8d99185e84634ea6f68ea702ee516a2b"
server_key_version: "encryptionkey@0"
I20260926 16:48:48.492093 28790 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260926 16:48:48.497761 28805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:48.500098 28790 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.002s
I20260926 16:48:48.500365 28790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "e5e43bb1ffd94cbd9ca290add2b4065b"
format_stamp: "Formatted at 2026-09-26 16:48:48 on dist-test-slave-wt2j"
server_key: "eeb0420774e0c4d8ded1fff24b55c373"
server_key_iv: "8d99185e84634ea6f68ea702ee516a2b"
server_key_version: "encryptionkey@0"
I20260926 16:48:48.500669 28790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:48:48.517616 28790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:48:48.523223 28790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:48:48.523686 28790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:48:48.525017 28790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:48:48.526823 28790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:48:48.526945 28790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:48.527114 28790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:48:48.527195 28790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:48.574631 28790 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:41419
I20260926 16:48:48.574747 28917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:41419 every 8 connection(s)
I20260926 16:48:48.576440 28790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:48:48.580963 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 28790
I20260926 16:48:48.581190 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance
I20260926 16:48:48.581755 10171 external_mini_cluster.cc:1428] Setting key c49a682d5ecaeef2f4fbd5d8617fe959
Sep 26 16:48:48 dist-test-slave-wt2j krb5kdc[28416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441328, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:48:48.609891 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:48.579767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:53515 (local address 127.9.238.254:46695)
0926 16:48:48.580394 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:48.580401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:48.581745 (+ 1344us) server_negotiation.cc:407] Connection header received
0926 16:48:48.584507 (+ 2762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:48.584514 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:48.584653 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:48.584794 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:48.586920 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:48.587906 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:48.589669 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:48.590103 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:48.590249 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:48.602478 (+ 12229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:48.602484 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:48.602489 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:48:48.602539 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:48.605804 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:48.607015 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:48.607020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:48.607023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:48.607188 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:48.607739 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:48.607743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:48.607745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:48.608095 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:48.608257 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:48.609181 (+ 924us) server_negotiation.cc:299] Negotiation successful
0926 16:48:48.609457 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":254,"threads_started":1}
I20260926 16:48:48.611035 28921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:48.580433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:46695 (local address 127.9.238.195:53515)
0926 16:48:48.581512 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:48.581566 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:48.584152 (+ 2586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:48.585160 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:48.585177 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:48.585940 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:48:48.586697 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:48.586719 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:48.588080 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:48.588084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:48.589506 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:48.589519 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:48.590378 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:48.590385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:48.590659 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:48.599094 (+ 8435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:48:48.599135 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:48:48.602160 (+ 3025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:48:48.606185 (+ 4025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:48:48.606194 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:48:48.606227 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:48:48.606858 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:48:48.607302 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:48:48.607306 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:48:48.607308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:48:48.607591 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:48:48.608272 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:48:48.608279 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:48:48.608720 (+ 441us) client_negotiation.cc:769] Sending connection context
0926 16:48:48.608991 (+ 271us) client_negotiation.cc:240] Negotiation successful
0926 16:48:48.609584 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":405,"threads_started":1}
I20260926 16:48:48.613276 28918 heartbeater.cc:347] Connected to a master server at 127.9.238.254:46695
I20260926 16:48:48.613693 28918 heartbeater.cc:464] Registering TS with master...
I20260926 16:48:48.614601 28918 heartbeater.cc:511] Master 127.9.238.254:46695 requested a full tablet report, sending...
I20260926 16:48:48.616724 28461 ts_manager.cc:194] Registered new tserver with Master: e5e43bb1ffd94cbd9ca290add2b4065b (127.9.238.195:41419)
I20260926 16:48:48.617740 28461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.195@KRBTEST.COM'} at 127.9.238.195:53515
I20260926 16:48:48.625072 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:48:48.647598 28920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:48.630309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53528 (local address 127.9.238.254:46695)
0926 16:48:48.630666 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:48.630676 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:48.631271 (+ 595us) server_negotiation.cc:407] Connection header received
0926 16:48:48.631509 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:48.631515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:48.631691 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:48.631827 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:48.633863 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:48.634708 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:48.636842 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:48.637348 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:48.637512 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:48.640921 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:48.640927 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:48.640931 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:48:48.640980 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:48.644106 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:48.645097 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:48.645107 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:48.645110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:48.645245 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:48.645745 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:48.645750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:48.645756 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:48.646182 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:48.646506 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:48.646960 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 16:48:48.647187 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260926 16:48:48.652698 28461 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:48:48.660422 28461 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:48:48.702839 28932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:48.676530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:41419 (local address 127.0.0.1:52962)
0926 16:48:48.690601 (+ 14071us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:48.690691 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:48.691033 (+ 342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:48.693127 (+ 2094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:48.693137 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:48.693191 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:48:48.693814 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:48.693843 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:48.696936 (+ 3093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:48.696951 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:48.699801 (+ 2850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:48.699819 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:48.701580 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:48.701593 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:48.701819 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:48.701881 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 16:48:48.702056 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 16:48:48.702316 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13679,"mutex_wait_us":103,"spinlock_wait_cycles":13312,"thread_start_us":13505,"threads_started":1}
I20260926 16:48:48.702798 28934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:48.677198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52962 (local address 127.9.238.195:41419)
0926 16:48:48.689537 (+ 12339us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:48.689549 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:48.690832 (+ 1283us) server_negotiation.cc:407] Connection header received
0926 16:48:48.691444 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:48.691455 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:48.692152 (+ 697us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:48.692598 (+ 446us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:48:48.694135 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:48.696645 (+ 2510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:48.700058 (+ 3413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:48.701342 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:48.701536 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:48.701846 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:48.702120 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 16:48:48.702323 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12106,"thread_start_us":12365,"threads_started":1}
I20260926 16:48:48.704257 28931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:48.676594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:38691 (local address 127.0.0.1:35266)
0926 16:48:48.690601 (+ 14007us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:48.690691 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:48.691033 (+ 342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:48.692366 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:48.692377 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:48.692423 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:48:48.693079 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:48.693107 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:48.695798 (+ 2691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:48.695807 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:48.699939 (+ 4132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:48.699970 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:48.703235 (+ 3265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:48.703242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:48.703459 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:48.703516 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 16:48:48.703631 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 16:48:48.703789 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13707,"mutex_wait_us":11,"spinlock_wait_cycles":18944,"thread_start_us":13478,"threads_started":1}
I20260926 16:48:48.704486 28933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:48.677197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35266 (local address 127.9.238.193:38691)
0926 16:48:48.689557 (+ 12360us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:48.689566 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:48.690832 (+ 1266us) server_negotiation.cc:407] Connection header received
0926 16:48:48.691415 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:48.691425 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:48.691748 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:48.691985 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:48:48.693386 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:48.695503 (+ 2117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:48.700237 (+ 4734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:48.702426 (+ 2189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:48.702640 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:48.702980 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:48.703770 (+ 790us) server_negotiation.cc:299] Negotiation successful
0926 16:48:48.703995 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12188,"thread_start_us":12365,"threads_started":1}
I20260926 16:48:48.708324 28853 tablet_service.cc:1467] Processing CreateTablet for tablet 7fed4d99da1641819bb1e0abd6a43ae7 (DEFAULT_TABLE table=test-table [id=38aab96c737f49fdaa6c6a104c9a5f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:48:48.709340 28585 tablet_service.cc:1467] Processing CreateTablet for tablet 7fed4d99da1641819bb1e0abd6a43ae7 (DEFAULT_TABLE table=test-table [id=38aab96c737f49fdaa6c6a104c9a5f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:48:48.710043 28853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fed4d99da1641819bb1e0abd6a43ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:48:48.710661 28585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fed4d99da1641819bb1e0abd6a43ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:48:48.713240 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:48.692173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34886 (local address 127.9.238.194:41071)
0926 16:48:48.706235 (+ 14062us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:48.706246 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:48.706281 (+ 35us) server_negotiation.cc:407] Connection header received
0926 16:48:48.706481 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:48.706492 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:48.706866 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:48.707105 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:48:48.707723 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:48.708856 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:48.710911 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:48.712043 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:48.712205 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:48.712445 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:48.712709 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 16:48:48.712897 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13860,"thread_start_us":13621,"threads_started":1}
I20260926 16:48:48.713240 28935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:48.691842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:41071 (local address 127.0.0.1:34886)
0926 16:48:48.698673 (+ 6831us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:48.698700 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:48.698858 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:48.707102 (+ 8244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:48.707108 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:48.707145 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:48:48.707491 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:48.707502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:48.709011 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:48.709018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:48.710682 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:48.710696 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:48.712283 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:48.712290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:48.712488 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:48.712529 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 16:48:48.712648 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 16:48:48.712811 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6637,"thread_start_us":7128,"threads_started":1}
I20260926 16:48:48.718212 28719 tablet_service.cc:1467] Processing CreateTablet for tablet 7fed4d99da1641819bb1e0abd6a43ae7 (DEFAULT_TABLE table=test-table [id=38aab96c737f49fdaa6c6a104c9a5f6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:48:48.719777 28719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fed4d99da1641819bb1e0abd6a43ae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:48:48.726137 28937 tablet_bootstrap.cc:492] T 7fed4d99da1641819bb1e0abd6a43ae7 P 7db29de44e1842ec80a015e99822f371: Bootstrap starting.
I20260926 16:48:48.730093 28938 tablet_bootstrap.cc:492] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b: Bootstrap starting.
I20260926 16:48:48.731088 28937 tablet_bootstrap.cc:654] T 7fed4d99da1641819bb1e0abd6a43ae7 P 7db29de44e1842ec80a015e99822f371: Neither blocks nor log segments found. Creating new log.
I20260926 16:48:48.732734 28937 log.cc:824] T 7fed4d99da1641819bb1e0abd6a43ae7 P 7db29de44e1842ec80a015e99822f371: Log is configured to *not* fsync() on all Append() calls
I20260926 16:48:48.733359 28938 tablet_bootstrap.cc:654] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b: Neither blocks nor log segments found. Creating new log.
I20260926 16:48:48.734705 28938 log.cc:824] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b: Log is configured to *not* fsync() on all Append() calls
I20260926 16:48:48.735984 28940 tablet_bootstrap.cc:492] T 7fed4d99da1641819bb1e0abd6a43ae7 P b8828e77745049dd936654ad51dc5a70: Bootstrap starting.
I20260926 16:48:48.736819 28937 tablet_bootstrap.cc:492] T 7fed4d99da1641819bb1e0abd6a43ae7 P 7db29de44e1842ec80a015e99822f371: No bootstrap required, opened a new log
I20260926 16:48:48.737155 28937 ts_tablet_manager.cc:1397] T 7fed4d99da1641819bb1e0abd6a43ae7 P 7db29de44e1842ec80a015e99822f371: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260926 16:48:48.738087 28938 tablet_bootstrap.cc:492] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b: No bootstrap required, opened a new log
I20260926 16:48:48.738358 28938 ts_tablet_manager.cc:1397] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260926 16:48:48.739328 28940 tablet_bootstrap.cc:654] T 7fed4d99da1641819bb1e0abd6a43ae7 P b8828e77745049dd936654ad51dc5a70: Neither blocks nor log segments found. Creating new log.
I20260926 16:48:48.740512 28940 log.cc:824] T 7fed4d99da1641819bb1e0abd6a43ae7 P b8828e77745049dd936654ad51dc5a70: Log is configured to *not* fsync() on all Append() calls
I20260926 16:48:48.743453 28940 tablet_bootstrap.cc:492] T 7fed4d99da1641819bb1e0abd6a43ae7 P b8828e77745049dd936654ad51dc5a70: No bootstrap required, opened a new log
I20260926 16:48:48.743778 28940 ts_tablet_manager.cc:1397] T 7fed4d99da1641819bb1e0abd6a43ae7 P b8828e77745049dd936654ad51dc5a70: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260926 16:48:48.749053 28937 raft_consensus.cc:348] T 7fed4d99da1641819bb1e0abd6a43ae7 P 7db29de44e1842ec80a015e99822f371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e43bb1ffd94cbd9ca290add2b4065b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 41419 } } peers { permanent_uuid: "7db29de44e1842ec80a015e99822f371" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 38691 } } peers { permanent_uuid: "b8828e77745049dd936654ad51dc5a70" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 41071 } }
I20260926 16:48:48.749392 28938 raft_consensus.cc:348] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e43bb1ffd94cbd9ca290add2b4065b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 41419 } } peers { permanent_uuid: "7db29de44e1842ec80a015e99822f371" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 38691 } } peers { permanent_uuid: "b8828e77745049dd936654ad51dc5a70" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 41071 } }
I20260926 16:48:48.749696 28937 raft_consensus.cc:374] T 7fed4d99da1641819bb1e0abd6a43ae7 P 7db29de44e1842ec80a015e99822f371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:48:48.749806 28937 raft_consensus.cc:729] T 7fed4d99da1641819bb1e0abd6a43ae7 P 7db29de44e1842ec80a015e99822f371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7db29de44e1842ec80a015e99822f371, State: Initialized, Role: FOLLOWER
I20260926 16:48:48.749826 28938 raft_consensus.cc:374] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:48:48.749930 28938 raft_consensus.cc:729] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e43bb1ffd94cbd9ca290add2b4065b, State: Initialized, Role: FOLLOWER
I20260926 16:48:48.750625 28937 consensus_queue.cc:260] T 7fed4d99da1641819bb1e0abd6a43ae7 P 7db29de44e1842ec80a015e99822f371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e43bb1ffd94cbd9ca290add2b4065b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 41419 } } peers { permanent_uuid: "7db29de44e1842ec80a015e99822f371" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 38691 } } peers { permanent_uuid: "b8828e77745049dd936654ad51dc5a70" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 41071 } }
I20260926 16:48:48.750625 28938 consensus_queue.cc:260] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e43bb1ffd94cbd9ca290add2b4065b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 41419 } } peers { permanent_uuid: "7db29de44e1842ec80a015e99822f371" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 38691 } } peers { permanent_uuid: "b8828e77745049dd936654ad51dc5a70" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 41071 } }
I20260926 16:48:48.752035 28918 heartbeater.cc:503] Master 127.9.238.254:46695 was elected leader, sending a full tablet report...
I20260926 16:48:48.752856 28940 raft_consensus.cc:348] T 7fed4d99da1641819bb1e0abd6a43ae7 P b8828e77745049dd936654ad51dc5a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e43bb1ffd94cbd9ca290add2b4065b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 41419 } } peers { permanent_uuid: "7db29de44e1842ec80a015e99822f371" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 38691 } } peers { permanent_uuid: "b8828e77745049dd936654ad51dc5a70" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 41071 } }
I20260926 16:48:48.753181 28937 ts_tablet_manager.cc:1428] T 7fed4d99da1641819bb1e0abd6a43ae7 P 7db29de44e1842ec80a015e99822f371: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260926 16:48:48.753402 28940 raft_consensus.cc:374] T 7fed4d99da1641819bb1e0abd6a43ae7 P b8828e77745049dd936654ad51dc5a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:48:48.753556 28940 raft_consensus.cc:729] T 7fed4d99da1641819bb1e0abd6a43ae7 P b8828e77745049dd936654ad51dc5a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8828e77745049dd936654ad51dc5a70, State: Initialized, Role: FOLLOWER
I20260926 16:48:48.754093 28938 ts_tablet_manager.cc:1428] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
I20260926 16:48:48.754456 28940 consensus_queue.cc:260] T 7fed4d99da1641819bb1e0abd6a43ae7 P b8828e77745049dd936654ad51dc5a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e43bb1ffd94cbd9ca290add2b4065b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 41419 } } peers { permanent_uuid: "7db29de44e1842ec80a015e99822f371" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 38691 } } peers { permanent_uuid: "b8828e77745049dd936654ad51dc5a70" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 41071 } }
I20260926 16:48:48.755766 28784 heartbeater.cc:503] Master 127.9.238.254:46695 was elected leader, sending a full tablet report...
I20260926 16:48:48.756412 28940 ts_tablet_manager.cc:1428] T 7fed4d99da1641819bb1e0abd6a43ae7 P b8828e77745049dd936654ad51dc5a70: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
W20260926 16:48:48.828770 28919 tablet.cc:2369] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:48:48.830715 28651 tablet.cc:2369] T 7fed4d99da1641819bb1e0abd6a43ae7 P 7db29de44e1842ec80a015e99822f371: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:48:48.893705 28944 raft_consensus.cc:482] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:48:48.894178 28944 raft_consensus.cc:504] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e43bb1ffd94cbd9ca290add2b4065b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 41419 } } peers { permanent_uuid: "7db29de44e1842ec80a015e99822f371" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 38691 } } peers { permanent_uuid: "b8828e77745049dd936654ad51dc5a70" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 41071 } }
I20260926 16:48:48.896477 28944 leader_election.cc:288] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7db29de44e1842ec80a015e99822f371 (127.9.238.193:38691), b8828e77745049dd936654ad51dc5a70 (127.9.238.194:41071)
I20260926 16:48:48.904734 28933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:48.897119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:37257 (local address 127.9.238.193:38691)
0926 16:48:48.897353 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:48.897360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:48.897392 (+ 32us) server_negotiation.cc:407] Connection header received
0926 16:48:48.897664 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:48.897669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:48.897745 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:48.897963 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:48:48.898707 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:48.899967 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:48.902249 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:48.903695 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:48.903841 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:48.903903 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:48.904327 (+ 424us) server_negotiation.cc:299] Negotiation successful
0926 16:48:48.904490 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 16:48:48.904914 28921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:48.897005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:38691 (local address 127.9.238.195:37257)
0926 16:48:48.897294 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:48.897319 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:48.897465 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:48.898198 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:48.898203 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:48.898236 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:48:48.898472 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:48.898487 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:48.900159 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:48.900165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:48.902060 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:48.902076 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:48.903889 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:48.903898 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:48.904112 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:48.904161 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 16:48:48.904279 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 16:48:48.904481 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":1}
I20260926 16:48:48.906258 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:48.897786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:43671 (local address 127.9.238.194:41071)
0926 16:48:48.898120 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:48.898126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:48.898553 (+ 427us) server_negotiation.cc:407] Connection header received
0926 16:48:48.898703 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:48.898710 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:48.898813 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:48.898964 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:48:48.899478 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:48.900636 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:48.903692 (+ 3056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:48.905097 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:48.905287 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:48.905403 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:48.905810 (+ 407us) server_negotiation.cc:299] Negotiation successful
0926 16:48:48.905966 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 16:48:48.906318 28946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:48.897529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:41071 (local address 127.9.238.195:43671)
0926 16:48:48.898388 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:48.898415 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:48.898554 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:48.898975 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:48.898978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:48.898997 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:48:48.899296 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:48.899303 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:48.900849 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:48.900855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:48.903353 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:48.903403 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:48.905392 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:48.905398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:48.905637 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:48.905664 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 16:48:48.905776 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 16:48:48.905965 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":410,"threads_started":1}
I20260926 16:48:48.906600 28605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fed4d99da1641819bb1e0abd6a43ae7" candidate_uuid: "e5e43bb1ffd94cbd9ca290add2b4065b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7db29de44e1842ec80a015e99822f371" is_pre_election: true
I20260926 16:48:48.907313 28605 raft_consensus.cc:2454] T 7fed4d99da1641819bb1e0abd6a43ae7 P 7db29de44e1842ec80a015e99822f371 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e43bb1ffd94cbd9ca290add2b4065b in term 0.
I20260926 16:48:48.907532 28739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fed4d99da1641819bb1e0abd6a43ae7" candidate_uuid: "e5e43bb1ffd94cbd9ca290add2b4065b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8828e77745049dd936654ad51dc5a70" is_pre_election: true
I20260926 16:48:48.908133 28739 raft_consensus.cc:2454] T 7fed4d99da1641819bb1e0abd6a43ae7 P b8828e77745049dd936654ad51dc5a70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e43bb1ffd94cbd9ca290add2b4065b in term 0.
I20260926 16:48:48.908142 28807 leader_election.cc:302] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7db29de44e1842ec80a015e99822f371, e5e43bb1ffd94cbd9ca290add2b4065b; no voters:
I20260926 16:48:48.908660 28944 raft_consensus.cc:2781] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:48:48.908828 28944 raft_consensus.cc:482] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:48:48.908979 28944 raft_consensus.cc:3037] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:48:48.911527 28944 raft_consensus.cc:504] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e43bb1ffd94cbd9ca290add2b4065b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 41419 } } peers { permanent_uuid: "7db29de44e1842ec80a015e99822f371" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 38691 } } peers { permanent_uuid: "b8828e77745049dd936654ad51dc5a70" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 41071 } }
I20260926 16:48:48.912384 28944 leader_election.cc:288] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b [CANDIDATE]: Term 1 election: Requested vote from peers 7db29de44e1842ec80a015e99822f371 (127.9.238.193:38691), b8828e77745049dd936654ad51dc5a70 (127.9.238.194:41071)
I20260926 16:48:48.913002 28605 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fed4d99da1641819bb1e0abd6a43ae7" candidate_uuid: "e5e43bb1ffd94cbd9ca290add2b4065b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7db29de44e1842ec80a015e99822f371"
I20260926 16:48:48.913105 28739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fed4d99da1641819bb1e0abd6a43ae7" candidate_uuid: "e5e43bb1ffd94cbd9ca290add2b4065b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8828e77745049dd936654ad51dc5a70"
I20260926 16:48:48.913300 28605 raft_consensus.cc:3037] T 7fed4d99da1641819bb1e0abd6a43ae7 P 7db29de44e1842ec80a015e99822f371 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:48:48.913457 28739 raft_consensus.cc:3037] T 7fed4d99da1641819bb1e0abd6a43ae7 P b8828e77745049dd936654ad51dc5a70 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:48:48.916077 28605 raft_consensus.cc:2454] T 7fed4d99da1641819bb1e0abd6a43ae7 P 7db29de44e1842ec80a015e99822f371 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e43bb1ffd94cbd9ca290add2b4065b in term 1.
I20260926 16:48:48.916139 28739 raft_consensus.cc:2454] T 7fed4d99da1641819bb1e0abd6a43ae7 P b8828e77745049dd936654ad51dc5a70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e43bb1ffd94cbd9ca290add2b4065b in term 1.
I20260926 16:48:48.916600 28807 leader_election.cc:302] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7db29de44e1842ec80a015e99822f371, e5e43bb1ffd94cbd9ca290add2b4065b; no voters:
I20260926 16:48:48.917114 28944 raft_consensus.cc:2781] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:48:48.917708 28944 raft_consensus.cc:686] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b [term 1 LEADER]: Becoming Leader. State: Replica: e5e43bb1ffd94cbd9ca290add2b4065b, State: Running, Role: LEADER
I20260926 16:48:48.918421 28944 consensus_queue.cc:237] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e43bb1ffd94cbd9ca290add2b4065b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 41419 } } peers { permanent_uuid: "7db29de44e1842ec80a015e99822f371" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 38691 } } peers { permanent_uuid: "b8828e77745049dd936654ad51dc5a70" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 41071 } }
W20260926 16:48:48.922541 28785 tablet.cc:2369] T 7fed4d99da1641819bb1e0abd6a43ae7 P b8828e77745049dd936654ad51dc5a70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:48:48.924865 28461 catalog_manager.cc:5495] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b reported cstate change: term changed from 0 to 1, leader changed from <none> to e5e43bb1ffd94cbd9ca290add2b4065b (127.9.238.195). New cstate: current_term: 1 leader_uuid: "e5e43bb1ffd94cbd9ca290add2b4065b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5e43bb1ffd94cbd9ca290add2b4065b" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 41419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7db29de44e1842ec80a015e99822f371" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 38691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8828e77745049dd936654ad51dc5a70" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 41071 } health_report { overall_health: UNKNOWN } } }
I20260926 16:48:48.963320 28934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:48.957163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52968 (local address 127.9.238.195:41419)
0926 16:48:48.957435 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:48.957441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:48.957482 (+ 41us) server_negotiation.cc:407] Connection header received
0926 16:48:48.957711 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:48.957716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:48.957797 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:48.957934 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:48:48.958330 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:48.959154 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:48.960762 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:48.961161 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:48.961269 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:48.961906 (+ 637us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:48:48.962523 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:48:48.962677 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:48.962844 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 16:48:48.962920 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 16:48:48.973625 28739 raft_consensus.cc:1261] T 7fed4d99da1641819bb1e0abd6a43ae7 P b8828e77745049dd936654ad51dc5a70 [term 1 FOLLOWER]: Refusing update from remote peer e5e43bb1ffd94cbd9ca290add2b4065b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:48:48.973696 28605 raft_consensus.cc:1261] T 7fed4d99da1641819bb1e0abd6a43ae7 P 7db29de44e1842ec80a015e99822f371 [term 1 FOLLOWER]: Refusing update from remote peer e5e43bb1ffd94cbd9ca290add2b4065b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:48:48.974911 28944 consensus_queue.cc:1035] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8828e77745049dd936654ad51dc5a70" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 41071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:48:48.975298 28947 consensus_queue.cc:1035] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7db29de44e1842ec80a015e99822f371" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 38691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:48:48.989948 28950 mvcc.cc:205] Tried to move back new op lower bound from 7333647683462610944 to 7333647683255734272. Current Snapshot: MvccSnapshot[applied={T|T < 7333647683462610944}]
I20260926 16:48:49.002326 28951 mvcc.cc:205] Tried to move back new op lower bound from 7333647683462610944 to 7333647683255734272. Current Snapshot: MvccSnapshot[applied={T|T < 7333647683462610944}]
I20260926 16:48:49.022769 28461 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:48:49.023108 28461 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:48:49.028122 28461 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9376ad89cd144f4d876330f8d435a3f7: Sending DeleteTablet for 3 replicas of tablet 7fed4d99da1641819bb1e0abd6a43ae7
I20260926 16:48:49.029778 28719 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fed4d99da1641819bb1e0abd6a43ae7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:48:49 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:34886
I20260926 16:48:49.029855 28585 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fed4d99da1641819bb1e0abd6a43ae7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:48:49 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:35266
I20260926 16:48:49.031243 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 28522
I20260926 16:48:49.031351 28853 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fed4d99da1641819bb1e0abd6a43ae7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:48:49 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:52962
I20260926 16:48:49.032330 28962 tablet_replica.cc:330] T 7fed4d99da1641819bb1e0abd6a43ae7 P 7db29de44e1842ec80a015e99822f371: stopping tablet replica
I20260926 16:48:49.032976 28962 raft_consensus.cc:2229] T 7fed4d99da1641819bb1e0abd6a43ae7 P 7db29de44e1842ec80a015e99822f371 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:48:49.033658 28962 raft_consensus.cc:2258] T 7fed4d99da1641819bb1e0abd6a43ae7 P 7db29de44e1842ec80a015e99822f371 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:48:49.035737 28963 tablet_replica.cc:330] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b: stopping tablet replica
I20260926 16:48:49.036242 28963 raft_consensus.cc:2229] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:48:49.037034 28963 raft_consensus.cc:2258] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:48:49.037639 28962 ts_tablet_manager.cc:1905] T 7fed4d99da1641819bb1e0abd6a43ae7 P 7db29de44e1842ec80a015e99822f371: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:48:49.037971 28961 tablet_replica.cc:330] T 7fed4d99da1641819bb1e0abd6a43ae7 P b8828e77745049dd936654ad51dc5a70: stopping tablet replica
I20260926 16:48:49.038439 28961 raft_consensus.cc:2229] T 7fed4d99da1641819bb1e0abd6a43ae7 P b8828e77745049dd936654ad51dc5a70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:48:49.039059 28961 raft_consensus.cc:2258] T 7fed4d99da1641819bb1e0abd6a43ae7 P b8828e77745049dd936654ad51dc5a70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:48:49.040268 28963 ts_tablet_manager.cc:1905] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:48:49.041323 28961 ts_tablet_manager.cc:1905] T 7fed4d99da1641819bb1e0abd6a43ae7 P b8828e77745049dd936654ad51dc5a70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:48:49.045452 28962 ts_tablet_manager.cc:1918] T 7fed4d99da1641819bb1e0abd6a43ae7 P 7db29de44e1842ec80a015e99822f371: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:48:49.045671 28962 log.cc:1196] T 7fed4d99da1641819bb1e0abd6a43ae7 P 7db29de44e1842ec80a015e99822f371: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-0/wal/wals/7fed4d99da1641819bb1e0abd6a43ae7
I20260926 16:48:49.046262 28962 ts_tablet_manager.cc:1939] T 7fed4d99da1641819bb1e0abd6a43ae7 P 7db29de44e1842ec80a015e99822f371: Deleting consensus metadata
I20260926 16:48:49.048592 28446 catalog_manager.cc:4841] TS 7db29de44e1842ec80a015e99822f371 (127.9.238.193:38691): tablet 7fed4d99da1641819bb1e0abd6a43ae7 (table test-table [id=38aab96c737f49fdaa6c6a104c9a5f6b]) successfully deleted
I20260926 16:48:49.050361 28961 ts_tablet_manager.cc:1918] T 7fed4d99da1641819bb1e0abd6a43ae7 P b8828e77745049dd936654ad51dc5a70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:48:49.050606 28961 log.cc:1196] T 7fed4d99da1641819bb1e0abd6a43ae7 P b8828e77745049dd936654ad51dc5a70: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-1/wal/wals/7fed4d99da1641819bb1e0abd6a43ae7
I20260926 16:48:49.051265 28961 ts_tablet_manager.cc:1939] T 7fed4d99da1641819bb1e0abd6a43ae7 P b8828e77745049dd936654ad51dc5a70: Deleting consensus metadata
Sep 26 16:48:49 dist-test-slave-wt2j krb5kdc[28416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.193@KRBTEST.COM
I20260926 16:48:49.053752 28446 catalog_manager.cc:4841] TS b8828e77745049dd936654ad51dc5a70 (127.9.238.194:41071): tablet 7fed4d99da1641819bb1e0abd6a43ae7 (table test-table [id=38aab96c737f49fdaa6c6a104c9a5f6b]) successfully deleted
I20260926 16:48:49.054649 28963 ts_tablet_manager.cc:1918] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:48:49.054859 28963 log.cc:1196] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790440988577176-10171-0/minicluster-data/ts-2/wal/wals/7fed4d99da1641819bb1e0abd6a43ae7
I20260926 16:48:49.055429 28963 ts_tablet_manager.cc:1939] T 7fed4d99da1641819bb1e0abd6a43ae7 P e5e43bb1ffd94cbd9ca290add2b4065b: Deleting consensus metadata
I20260926 16:48:49.057605 28445 catalog_manager.cc:4841] TS e5e43bb1ffd94cbd9ca290add2b4065b (127.9.238.195:41419): tablet 7fed4d99da1641819bb1e0abd6a43ae7 (table test-table [id=38aab96c737f49fdaa6c6a104c9a5f6b]) successfully deleted
I20260926 16:48:49.061594 28933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:49.032902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35276 (local address 127.9.238.193:38691)
0926 16:48:49.033503 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:49.033511 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:49.035205 (+ 1694us) server_negotiation.cc:407] Connection header received
0926 16:48:49.035575 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:49.035586 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:49.035712 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:49.035804 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:49.040911 (+ 5107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:49.042104 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:49.046499 (+ 4395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:49.047231 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:49.047359 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:49.054288 (+ 6929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:49.054309 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:49.054327 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:48:49.054389 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:49.058302 (+ 3913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:49.059143 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:49.059152 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:49.059155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:49.059305 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:49.059804 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:49.059809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:49.059812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:49.060233 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:49.060579 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:49.060902 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 16:48:49.061148 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461}
I20260926 16:48:49.062372 28645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:48:49.197074 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 28522
I20260926 16:48:49.225725 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 28656
Sep 26 16:48:49 dist-test-slave-wt2j krb5kdc[28416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.194@KRBTEST.COM
I20260926 16:48:49.246385 28936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:49.226853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34900 (local address 127.9.238.194:41071)
0926 16:48:49.227061 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:49.227066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:49.227088 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:48:49.227169 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:49.227174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:49.227264 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:49.227369 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:49.229143 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:49.230005 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:49.232341 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:49.232783 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:49.232904 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:49.238779 (+ 5875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:49.238803 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:49.238825 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:48:49.238884 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:49.242393 (+ 3509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:49.243506 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:49.243517 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:49.243520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:49.243798 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:49.244357 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:49.244362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:49.244364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:49.244882 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:49.245167 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:49.245652 (+ 485us) server_negotiation.cc:299] Negotiation successful
0926 16:48:49.245905 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 16:48:49.247404 28779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:48:49.351131 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 28656
I20260926 16:48:49.376727 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 28790
Sep 26 16:48:49 dist-test-slave-wt2j krb5kdc[28416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.195@KRBTEST.COM
I20260926 16:48:49.397128 28934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:49.377931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52972 (local address 127.9.238.195:41419)
0926 16:48:49.378293 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:49.378300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:49.378329 (+ 29us) server_negotiation.cc:407] Connection header received
0926 16:48:49.378412 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:49.378420 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:49.378526 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:49.378672 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:49.380188 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:49.381464 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:49.382896 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:49.383861 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:49.384018 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:49.389040 (+ 5022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:49.389066 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:49.389094 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:48:49.389141 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:49.393598 (+ 4457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:49.394500 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:49.394511 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:49.394515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:49.394713 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:49.395170 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:49.395176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:49.395180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:49.395731 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:49.396085 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:49.396466 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 16:48:49.396717 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260926 16:48:49.398223 28913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:48:49.530507 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 28790
I20260926 16:48:49.552802 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 28429
I20260926 16:48:49.554195 28491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:48:49.686683 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 28429
2026-09-26T16:48:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (5118 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T16:48:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:48:49Z Disabled control of system clock
I20260926 16:48:49.778271 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:43317
--webserver_interface=127.9.238.254
--webserver_port=0
--builtin_ntp_servers=127.9.238.212:46519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:43317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 16:48:50.151187 28972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:48:50.151516 28972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:48:50.151678 28972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:48:50.159152 28972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:48:50.159278 28972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:48:50.159317 28972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:48:50.159369 28972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:48:50.159407 28972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:48:50.168975 28972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:46519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:43317
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:43317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:48:50.170372 28972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:48:50.172204 28972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:48:50.185437 28977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:50.185846 28978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:50.185900 28980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:48:50.187783 28972 server_base.cc:1034] running on GCE node
I20260926 16:48:50.188889 28972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:48:50.190778 28972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:48:50.192023 28972 hybrid_clock.cc:648] HybridClock initialized: now 1790441330191972 us; error 40 us; skew 500 ppm
I20260926 16:48:50.192514 28972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:48:50.195623 28972 webserver.cc:460] Webserver started at http://127.9.238.254:44041/ using document root <none> and password file <none>
I20260926 16:48:50.196587 28972 fs_manager.cc:359] Metadata directory not provided
I20260926 16:48:50.196691 28972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:48:50.197137 28972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:48:50.201028 28972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "ee29643ab6c84ad9b47f1d9132db07d2"
format_stamp: "Formatted at 2026-09-26 16:48:50 on dist-test-slave-wt2j"
server_key: "e8527203e0d97539dbd8c5aeda4b54ad"
server_key_iv: "1ffbabe1c146d02e5107bacec4c5b62f"
server_key_version: "encryptionkey@0"
I20260926 16:48:50.202675 28972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "ee29643ab6c84ad9b47f1d9132db07d2"
format_stamp: "Formatted at 2026-09-26 16:48:50 on dist-test-slave-wt2j"
server_key: "e8527203e0d97539dbd8c5aeda4b54ad"
server_key_iv: "1ffbabe1c146d02e5107bacec4c5b62f"
server_key_version: "encryptionkey@0"
I20260926 16:48:50.209851 28972 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260926 16:48:50.214921 28986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:50.217068 28972 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:48:50.217386 28972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "ee29643ab6c84ad9b47f1d9132db07d2"
format_stamp: "Formatted at 2026-09-26 16:48:50 on dist-test-slave-wt2j"
server_key: "e8527203e0d97539dbd8c5aeda4b54ad"
server_key_iv: "1ffbabe1c146d02e5107bacec4c5b62f"
server_key_version: "encryptionkey@0"
I20260926 16:48:50.217707 28972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:48:50.245270 28972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:48:50.246646 28972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:48:50.247040 28972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:48:50.272044 28972 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:43317
I20260926 16:48:50.272073 29037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:43317 every 8 connection(s)
I20260926 16:48:50.273940 28972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:48:50.277345 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 28972
I20260926 16:48:50.277559 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:48:50.278098 10171 external_mini_cluster.cc:1428] Setting key c2785829caf35f13f1f2ef84f0617e87
I20260926 16:48:50.278910 29038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:48:50.295043 29038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee29643ab6c84ad9b47f1d9132db07d2: Bootstrap starting.
I20260926 16:48:50.296729 29041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:50.280582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40316 (local address 127.9.238.254:43317)
0926 16:48:50.281905 (+ 1323us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:50.281919 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:50.281979 (+ 60us) server_negotiation.cc:407] Connection header received
0926 16:48:50.284505 (+ 2526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:50.284539 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:50.285252 (+ 713us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:50.285766 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:50.286603 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:50.288386 (+ 1783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:50.290510 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:50.291638 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:50.291876 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:50.293393 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:50.293403 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:50.293423 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:48:50.294146 (+ 723us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:50.294249 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:48:50.294276 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:50.294414 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:50.294738 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 16:48:50.295331 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":747,"thread_start_us":480,"threads_started":1}
I20260926 16:48:50.298704 29038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee29643ab6c84ad9b47f1d9132db07d2: Neither blocks nor log segments found. Creating new log.
I20260926 16:48:50.299969 29038 log.cc:824] T 00000000000000000000000000000000 P ee29643ab6c84ad9b47f1d9132db07d2: Log is configured to *not* fsync() on all Append() calls
I20260926 16:48:50.303282 29038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee29643ab6c84ad9b47f1d9132db07d2: No bootstrap required, opened a new log
I20260926 16:48:50.311268 29038 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee29643ab6c84ad9b47f1d9132db07d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee29643ab6c84ad9b47f1d9132db07d2" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 43317 } }
I20260926 16:48:50.311754 29038 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee29643ab6c84ad9b47f1d9132db07d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:48:50.311843 29038 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee29643ab6c84ad9b47f1d9132db07d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee29643ab6c84ad9b47f1d9132db07d2, State: Initialized, Role: FOLLOWER
I20260926 16:48:50.312374 29038 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee29643ab6c84ad9b47f1d9132db07d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee29643ab6c84ad9b47f1d9132db07d2" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 43317 } }
I20260926 16:48:50.312572 29038 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee29643ab6c84ad9b47f1d9132db07d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:48:50.312675 29038 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee29643ab6c84ad9b47f1d9132db07d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:48:50.312796 29038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee29643ab6c84ad9b47f1d9132db07d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:48:50.314968 29038 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee29643ab6c84ad9b47f1d9132db07d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee29643ab6c84ad9b47f1d9132db07d2" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 43317 } }
I20260926 16:48:50.315492 29038 leader_election.cc:302] T 00000000000000000000000000000000 P ee29643ab6c84ad9b47f1d9132db07d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee29643ab6c84ad9b47f1d9132db07d2; no voters:
I20260926 16:48:50.316195 29038 leader_election.cc:288] T 00000000000000000000000000000000 P ee29643ab6c84ad9b47f1d9132db07d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:48:50.316526 29043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee29643ab6c84ad9b47f1d9132db07d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:48:50.317433 29043 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee29643ab6c84ad9b47f1d9132db07d2 [term 1 LEADER]: Becoming Leader. State: Replica: ee29643ab6c84ad9b47f1d9132db07d2, State: Running, Role: LEADER
I20260926 16:48:50.318089 29043 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee29643ab6c84ad9b47f1d9132db07d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee29643ab6c84ad9b47f1d9132db07d2" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 43317 } }
I20260926 16:48:50.318415 29038 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee29643ab6c84ad9b47f1d9132db07d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:48:50.322732 29045 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee29643ab6c84ad9b47f1d9132db07d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee29643ab6c84ad9b47f1d9132db07d2. Latest consensus state: current_term: 1 leader_uuid: "ee29643ab6c84ad9b47f1d9132db07d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee29643ab6c84ad9b47f1d9132db07d2" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 43317 } } }
I20260926 16:48:50.323446 29045 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee29643ab6c84ad9b47f1d9132db07d2 [sys.catalog]: This master's current role is: LEADER
I20260926 16:48:50.322698 29044 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee29643ab6c84ad9b47f1d9132db07d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee29643ab6c84ad9b47f1d9132db07d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee29643ab6c84ad9b47f1d9132db07d2" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 43317 } } }
I20260926 16:48:50.323587 29044 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee29643ab6c84ad9b47f1d9132db07d2 [sys.catalog]: This master's current role is: LEADER
I20260926 16:48:50.324676 29053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:48:50.332257 29053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:48:50.341298 29053 catalog_manager.cc:1347] Generated new cluster ID: 98b6f5345f7948d7a2f0679cb71c6881
I20260926 16:48:50.341441 29053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:48:50.366379 29053 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:48:50.368673 29053 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:48:50.383247 29053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee29643ab6c84ad9b47f1d9132db07d2: Generated new TSK 0
I20260926 16:48:50.384778 29053 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:48:50.402944 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:0
--local_ip_for_outbound_sockets=127.9.238.193
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:43317
--builtin_ntp_servers=127.9.238.212:46519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 16:48:50.751663 29062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:48:50.751996 29062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:48:50.752089 29062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:48:50.759819 29062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:48:50.759935 29062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:48:50.760053 29062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:48:50.769488 29062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:46519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:43317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:48:50.770761 29062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:48:50.772584 29062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:48:50.788700 29067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:50.788841 29070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:50.788702 29068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:48:50.790522 29062 server_base.cc:1034] running on GCE node
I20260926 16:48:50.791348 29062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:48:50.792595 29062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:48:50.793905 29062 hybrid_clock.cc:648] HybridClock initialized: now 1790441330793851 us; error 44 us; skew 500 ppm
I20260926 16:48:50.794330 29062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:48:50.797297 29062 webserver.cc:460] Webserver started at http://127.9.238.193:39433/ using document root <none> and password file <none>
I20260926 16:48:50.798161 29062 fs_manager.cc:359] Metadata directory not provided
I20260926 16:48:50.798312 29062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:48:50.798686 29062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:48:50.801975 29062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-0/data/instance:
uuid: "1a3714bdd55b428fb0654a39e5d385c0"
format_stamp: "Formatted at 2026-09-26 16:48:50 on dist-test-slave-wt2j"
server_key: "fe0c2e99f37bb5c50ecb2b13a61569e6"
server_key_iv: "03a7833868c2ae2482e1566d7595b868"
server_key_version: "encryptionkey@0"
I20260926 16:48:50.802866 29062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance:
uuid: "1a3714bdd55b428fb0654a39e5d385c0"
format_stamp: "Formatted at 2026-09-26 16:48:50 on dist-test-slave-wt2j"
server_key: "fe0c2e99f37bb5c50ecb2b13a61569e6"
server_key_iv: "03a7833868c2ae2482e1566d7595b868"
server_key_version: "encryptionkey@0"
I20260926 16:48:50.809052 29062 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260926 16:48:50.813606 29076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:50.815410 29062 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:48:50.815713 29062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "1a3714bdd55b428fb0654a39e5d385c0"
format_stamp: "Formatted at 2026-09-26 16:48:50 on dist-test-slave-wt2j"
server_key: "fe0c2e99f37bb5c50ecb2b13a61569e6"
server_key_iv: "03a7833868c2ae2482e1566d7595b868"
server_key_version: "encryptionkey@0"
I20260926 16:48:50.815971 29062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:48:50.836920 29062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:48:50.837814 29062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:48:50.838135 29062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:48:50.839250 29062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:48:50.841084 29062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:48:50.841226 29062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:50.841364 29062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:48:50.841454 29062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:50.880441 29062 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:41197
I20260926 16:48:50.880566 29188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:41197 every 8 connection(s)
I20260926 16:48:50.881970 29062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:48:50.890270 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 29062
I20260926 16:48:50.890554 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance
I20260926 16:48:50.891047 10171 external_mini_cluster.cc:1428] Setting key d42604b3d9519fef24e101398c3f43cc
I20260926 16:48:50.894665 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:0
--local_ip_for_outbound_sockets=127.9.238.194
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:43317
--builtin_ntp_servers=127.9.238.212:46519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 16:48:50.898104 29191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:50.885309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:37291 (local address 127.9.238.254:43317)
0926 16:48:50.885917 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:50.885925 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:50.887541 (+ 1616us) server_negotiation.cc:407] Connection header received
0926 16:48:50.889612 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:50.889618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:50.889767 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:50.889988 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:50.891265 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:50.892337 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:50.894400 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:50.895154 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:50.895322 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:50.896910 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:50.896918 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:50.896926 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:48:50.896958 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:50.897023 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:48:50.897046 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:50.897158 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:50.897418 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 16:48:50.897557 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":262,"threads_started":1}
I20260926 16:48:50.898816 29192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:50.886142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:43317 (local address 127.9.238.193:37291)
0926 16:48:50.887338 (+ 1196us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:50.887385 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:50.889295 (+ 1910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:50.890209 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:50.890219 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:50.890316 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:48:50.891025 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:50.891043 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:50.892557 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:50.892563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:50.894230 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:50.894243 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:50.895384 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:50.895389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:50.895667 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:50.896528 (+ 861us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:48:50.896530 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:48:50.896572 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:48:50.896662 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:48:50.897188 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:48:50.897196 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:48:50.897202 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 16:48:50.897360 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 16:48:50.897673 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":390,"threads_started":1}
I20260926 16:48:50.900617 29189 heartbeater.cc:347] Connected to a master server at 127.9.238.254:43317
I20260926 16:48:50.901106 29189 heartbeater.cc:464] Registering TS with master...
I20260926 16:48:50.902369 29189 heartbeater.cc:511] Master 127.9.238.254:43317 requested a full tablet report, sending...
I20260926 16:48:50.905205 29003 ts_manager.cc:194] Registered new tserver with Master: 1a3714bdd55b428fb0654a39e5d385c0 (127.9.238.193:41197)
I20260926 16:48:50.906800 29003 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.238.193:37291
W20260926 16:48:51.305374 29193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:48:51.305787 29193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:48:51.305898 29193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:48:51.314906 29193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:48:51.315063 29193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:48:51.315191 29193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:48:51.326380 29193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:46519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:43317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:48:51.328215 29193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:48:51.330497 29193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:48:51.347847 29198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:51.347920 29199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:51.347981 29201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:48:51.348800 29193 server_base.cc:1034] running on GCE node
I20260926 16:48:51.349673 29193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:48:51.351274 29193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:48:51.352578 29193 hybrid_clock.cc:648] HybridClock initialized: now 1790441331352498 us; error 70 us; skew 500 ppm
I20260926 16:48:51.353032 29193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:48:51.356190 29193 webserver.cc:460] Webserver started at http://127.9.238.194:43615/ using document root <none> and password file <none>
I20260926 16:48:51.357191 29193 fs_manager.cc:359] Metadata directory not provided
I20260926 16:48:51.357347 29193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:48:51.357764 29193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:48:51.361680 29193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-1/data/instance:
uuid: "ec66aae97d2a4bafb03391d535f27e6f"
format_stamp: "Formatted at 2026-09-26 16:48:51 on dist-test-slave-wt2j"
server_key: "e71ee554d388b34877631b0c92a8b673"
server_key_iv: "6c12767fca25af144f7c5e826aa2e291"
server_key_version: "encryptionkey@0"
I20260926 16:48:51.362676 29193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance:
uuid: "ec66aae97d2a4bafb03391d535f27e6f"
format_stamp: "Formatted at 2026-09-26 16:48:51 on dist-test-slave-wt2j"
server_key: "e71ee554d388b34877631b0c92a8b673"
server_key_iv: "6c12767fca25af144f7c5e826aa2e291"
server_key_version: "encryptionkey@0"
I20260926 16:48:51.369182 29193 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260926 16:48:51.373659 29207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:51.375425 29193 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260926 16:48:51.375677 29193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "ec66aae97d2a4bafb03391d535f27e6f"
format_stamp: "Formatted at 2026-09-26 16:48:51 on dist-test-slave-wt2j"
server_key: "e71ee554d388b34877631b0c92a8b673"
server_key_iv: "6c12767fca25af144f7c5e826aa2e291"
server_key_version: "encryptionkey@0"
I20260926 16:48:51.375957 29193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:48:51.391287 29193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:48:51.392565 29193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:48:51.392941 29193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:48:51.394250 29193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:48:51.396203 29193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:48:51.396353 29193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:51.396510 29193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:48:51.396606 29193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:51.439103 29193 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:37305
I20260926 16:48:51.439214 29319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:37305 every 8 connection(s)
I20260926 16:48:51.440845 29193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:48:51.443531 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 29193
I20260926 16:48:51.443819 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance
I20260926 16:48:51.444449 10171 external_mini_cluster.cc:1428] Setting key cd34cf7ef9a299625d493126b8829c59
I20260926 16:48:51.447978 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:0
--local_ip_for_outbound_sockets=127.9.238.195
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:43317
--builtin_ntp_servers=127.9.238.212:46519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 16:48:51.458436 29322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:51.444553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:41939 (local address 127.9.238.254:43317)
0926 16:48:51.445241 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:51.445247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:51.447653 (+ 2406us) server_negotiation.cc:407] Connection header received
0926 16:48:51.449600 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:51.449609 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:51.449712 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:51.449866 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:51.451315 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:51.452563 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:51.454675 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:51.455434 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:51.455662 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:51.457508 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:51.457517 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:51.457522 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:48:51.457546 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:51.457602 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:48:51.457616 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:51.457722 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:51.457976 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 16:48:51.458083 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":327,"threads_started":1}
I20260926 16:48:51.459769 29323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:51.445436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:43317 (local address 127.9.238.194:41939)
0926 16:48:51.446801 (+ 1365us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:51.446856 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:51.449035 (+ 2179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:51.450123 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:51.450134 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:51.450273 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:48:51.451089 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:51.451111 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:51.452777 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:51.452784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:51.454483 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:51.454500 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:51.455776 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:51.455783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:51.456050 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:51.457087 (+ 1037us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:48:51.457091 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:48:51.457155 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:48:51.457269 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:48:51.457724 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:48:51.457738 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:48:51.457749 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 16:48:51.457963 (+ 214us) client_negotiation.cc:240] Negotiation successful
0926 16:48:51.458488 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":430,"threads_started":1}
I20260926 16:48:51.461647 29320 heartbeater.cc:347] Connected to a master server at 127.9.238.254:43317
I20260926 16:48:51.462014 29320 heartbeater.cc:464] Registering TS with master...
I20260926 16:48:51.462845 29320 heartbeater.cc:511] Master 127.9.238.254:43317 requested a full tablet report, sending...
I20260926 16:48:51.464627 29003 ts_manager.cc:194] Registered new tserver with Master: ec66aae97d2a4bafb03391d535f27e6f (127.9.238.194:37305)
I20260926 16:48:51.465529 29003 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.238.194:41939
W20260926 16:48:51.869586 29324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:48:51.869975 29324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:48:51.870148 29324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:48:51.880173 29324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:48:51.880316 29324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:48:51.880481 29324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:48:51.893167 29324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:46519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:43317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:48:51.894517 29324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:48:51.896564 29324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 16:48:51.911330 29189 heartbeater.cc:503] Master 127.9.238.254:43317 was elected leader, sending a full tablet report...
W20260926 16:48:51.913224 29329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:51.914078 29330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:51.914129 29332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:48:51.914983 29324 server_base.cc:1034] running on GCE node
I20260926 16:48:51.916030 29324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:48:51.917500 29324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:48:51.918774 29324 hybrid_clock.cc:648] HybridClock initialized: now 1790441331918717 us; error 44 us; skew 500 ppm
I20260926 16:48:51.919226 29324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:48:51.922751 29324 webserver.cc:460] Webserver started at http://127.9.238.195:36909/ using document root <none> and password file <none>
I20260926 16:48:51.923918 29324 fs_manager.cc:359] Metadata directory not provided
I20260926 16:48:51.924077 29324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:48:51.924474 29324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:48:51.928125 29324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-2/data/instance:
uuid: "2320aef7748c43ca8f2b9d0682a1c1aa"
format_stamp: "Formatted at 2026-09-26 16:48:51 on dist-test-slave-wt2j"
server_key: "6df1f7fbe597c7d8aa30342f7fb46eaa"
server_key_iv: "dfd86a376d8ea09e8d66220a651a317f"
server_key_version: "encryptionkey@0"
I20260926 16:48:51.929136 29324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance:
uuid: "2320aef7748c43ca8f2b9d0682a1c1aa"
format_stamp: "Formatted at 2026-09-26 16:48:51 on dist-test-slave-wt2j"
server_key: "6df1f7fbe597c7d8aa30342f7fb46eaa"
server_key_iv: "dfd86a376d8ea09e8d66220a651a317f"
server_key_version: "encryptionkey@0"
I20260926 16:48:51.936913 29324 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260926 16:48:51.942615 29338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:51.944762 29324 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260926 16:48:51.945029 29324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "2320aef7748c43ca8f2b9d0682a1c1aa"
format_stamp: "Formatted at 2026-09-26 16:48:51 on dist-test-slave-wt2j"
server_key: "6df1f7fbe597c7d8aa30342f7fb46eaa"
server_key_iv: "dfd86a376d8ea09e8d66220a651a317f"
server_key_version: "encryptionkey@0"
I20260926 16:48:51.945325 29324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:48:51.964005 29324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:48:51.965102 29324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:48:51.965476 29324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:48:51.966756 29324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:48:51.968743 29324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:48:51.968890 29324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:51.969048 29324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:48:51.969141 29324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:52.020753 29324 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:46097
I20260926 16:48:52.020816 29450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:46097 every 8 connection(s)
I20260926 16:48:52.022739 29324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:48:52.024240 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 29324
I20260926 16:48:52.024447 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance
I20260926 16:48:52.025174 10171 external_mini_cluster.cc:1428] Setting key 47dbddd1cfbdedf2801a1e05559e4480
I20260926 16:48:52.039398 29453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:52.026616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:45575 (local address 127.9.238.254:43317)
0926 16:48:52.027227 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:52.027235 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:52.029194 (+ 1959us) server_negotiation.cc:407] Connection header received
0926 16:48:52.031289 (+ 2095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:52.031294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:52.031384 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:52.031529 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:52.033200 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.034017 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.036052 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.036542 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.036685 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:52.038345 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:52.038350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:52.038353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:48:52.038378 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:52.038425 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:48:52.038438 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:52.038578 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:52.038827 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 16:48:52.039030 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":293,"threads_started":1}
I20260926 16:48:52.040774 29454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:52.027656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:43317 (local address 127.9.238.195:45575)
0926 16:48:52.028964 (+ 1308us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:52.029025 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:52.030932 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:52.031946 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:52.031956 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:52.032089 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:48:52.032964 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:52.032987 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.034202 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.034206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:52.035817 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:52.035832 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.036796 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.036800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:52.037071 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:52.037919 (+ 848us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:48:52.037921 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:48:52.037974 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:48:52.038080 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:48:52.038555 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:48:52.038564 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:48:52.038572 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 16:48:52.038747 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 16:48:52.039262 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":407,"threads_started":1}
I20260926 16:48:52.042701 29451 heartbeater.cc:347] Connected to a master server at 127.9.238.254:43317
I20260926 16:48:52.043042 29451 heartbeater.cc:464] Registering TS with master...
I20260926 16:48:52.044056 29451 heartbeater.cc:511] Master 127.9.238.254:43317 requested a full tablet report, sending...
I20260926 16:48:52.045723 29003 ts_manager.cc:194] Registered new tserver with Master: 2320aef7748c43ca8f2b9d0682a1c1aa (127.9.238.195:46097)
I20260926 16:48:52.046634 29003 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.238.195:45575
I20260926 16:48:52.054193 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:48:52.065963 29453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:52.058859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44900 (local address 127.9.238.254:43317)
0926 16:48:52.059190 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:52.059197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:52.059608 (+ 411us) server_negotiation.cc:407] Connection header received
0926 16:48:52.059833 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:52.059838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:52.059951 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:52.060061 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:52.060658 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.061471 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.063024 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.063466 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.063664 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:52.064923 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:52.064929 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:52.064933 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:48:52.064959 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:52.065005 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:48:52.065019 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:52.065156 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:52.065394 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 16:48:52.065510 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260926 16:48:52.071345 29003 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:48:52.079180 29003 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:48:52.112680 29464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:52.095391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:37305 (local address 127.0.0.1:40544)
0926 16:48:52.096598 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:52.096662 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:52.096938 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:52.100799 (+ 3861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:52.100812 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:52.100868 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:48:52.101510 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:52.101529 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.104554 (+ 3025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.104564 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:52.108188 (+ 3624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:52.108222 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.111149 (+ 2927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.111160 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:52.111490 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:52.111605 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 16:48:52.111764 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 16:48:52.112086 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":376,"threads_started":1}
I20260926 16:48:52.114231 29467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:52.097148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53876 (local address 127.9.238.195:46097)
0926 16:48:52.098084 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:52.098097 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:52.100331 (+ 2234us) server_negotiation.cc:407] Connection header received
0926 16:48:52.100819 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:52.100834 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:52.101196 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:52.101471 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:48:52.103130 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.105365 (+ 2235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.109461 (+ 4096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.111716 (+ 2255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.111906 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:52.112379 (+ 473us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:52.113274 (+ 895us) server_negotiation.cc:299] Negotiation successful
0926 16:48:52.113616 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":430,"threads_started":1}
I20260926 16:48:52.114841 29465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:52.095593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40544 (local address 127.9.238.194:37305)
0926 16:48:52.100044 (+ 4451us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:52.100050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:52.100078 (+ 28us) server_negotiation.cc:407] Connection header received
0926 16:48:52.100204 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:52.100211 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:52.100504 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:52.101814 (+ 1310us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:48:52.102139 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.104326 (+ 2187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.108617 (+ 4291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.110778 (+ 2161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.113548 (+ 2770us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:52.113953 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:52.114068 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 16:48:52.114340 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4294,"thread_start_us":4164,"threads_started":1}
I20260926 16:48:52.115310 29466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:52.096799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:46097 (local address 127.0.0.1:53876)
0926 16:48:52.100167 (+ 3368us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:52.100192 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:52.100507 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:52.101958 (+ 1451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:52.101966 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:52.102005 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:48:52.102790 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:52.102812 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.105607 (+ 2795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.105612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:52.109065 (+ 3453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:52.109098 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.112704 (+ 3606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.112716 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:52.112980 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:52.113029 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 16:48:52.114612 (+ 1583us) client_negotiation.cc:240] Negotiation successful
0926 16:48:52.114883 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3176,"thread_start_us":3000,"threads_started":1}
I20260926 16:48:52.121423 29255 tablet_service.cc:1467] Processing CreateTablet for tablet 0f26235d3e464465b7adec3f462601c9 (DEFAULT_TABLE table=test-table [id=bdc598efeaf142f29111243528208446]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:48:52.121420 29386 tablet_service.cc:1467] Processing CreateTablet for tablet 0f26235d3e464465b7adec3f462601c9 (DEFAULT_TABLE table=test-table [id=bdc598efeaf142f29111243528208446]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:48:52.122598 29464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:52.100290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:41197 (local address 127.0.0.1:49182)
0926 16:48:52.113113 (+ 12823us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:52.113143 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:52.113305 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:52.116306 (+ 3001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:52.116311 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:52.116342 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:48:52.116565 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:52.116577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.117982 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.117987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:52.119667 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:52.119680 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.121694 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.121700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:52.121965 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:52.122004 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 16:48:52.122088 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 16:48:52.122254 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12628,"mutex_wait_us":22,"thread_start_us":15424,"threads_started":1}
I20260926 16:48:52.123493 29255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f26235d3e464465b7adec3f462601c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:48:52.123492 29386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f26235d3e464465b7adec3f462601c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:48:52.123859 29469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:52.100569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49182 (local address 127.9.238.193:41197)
0926 16:48:52.115729 (+ 15160us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:52.115736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:52.115763 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:48:52.115849 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:52.115855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:52.116080 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:52.116313 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:48:52.116723 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.117792 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.119822 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.121509 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.122876 (+ 1367us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:52.123169 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:52.123273 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 16:48:52.123473 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14957,"thread_start_us":14818,"threads_started":1}
I20260926 16:48:52.128366 29124 tablet_service.cc:1467] Processing CreateTablet for tablet 0f26235d3e464465b7adec3f462601c9 (DEFAULT_TABLE table=test-table [id=bdc598efeaf142f29111243528208446]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:48:52.129940 29124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f26235d3e464465b7adec3f462601c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:48:52.142927 29471 tablet_bootstrap.cc:492] T 0f26235d3e464465b7adec3f462601c9 P 2320aef7748c43ca8f2b9d0682a1c1aa: Bootstrap starting.
I20260926 16:48:52.144060 29470 tablet_bootstrap.cc:492] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f: Bootstrap starting.
I20260926 16:48:52.144443 29472 tablet_bootstrap.cc:492] T 0f26235d3e464465b7adec3f462601c9 P 1a3714bdd55b428fb0654a39e5d385c0: Bootstrap starting.
I20260926 16:48:52.148198 29471 tablet_bootstrap.cc:654] T 0f26235d3e464465b7adec3f462601c9 P 2320aef7748c43ca8f2b9d0682a1c1aa: Neither blocks nor log segments found. Creating new log.
I20260926 16:48:52.148895 29472 tablet_bootstrap.cc:654] T 0f26235d3e464465b7adec3f462601c9 P 1a3714bdd55b428fb0654a39e5d385c0: Neither blocks nor log segments found. Creating new log.
I20260926 16:48:52.149030 29470 tablet_bootstrap.cc:654] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f: Neither blocks nor log segments found. Creating new log.
I20260926 16:48:52.149758 29471 log.cc:824] T 0f26235d3e464465b7adec3f462601c9 P 2320aef7748c43ca8f2b9d0682a1c1aa: Log is configured to *not* fsync() on all Append() calls
I20260926 16:48:52.150369 29470 log.cc:824] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f: Log is configured to *not* fsync() on all Append() calls
I20260926 16:48:52.150355 29472 log.cc:824] T 0f26235d3e464465b7adec3f462601c9 P 1a3714bdd55b428fb0654a39e5d385c0: Log is configured to *not* fsync() on all Append() calls
I20260926 16:48:52.154052 29470 tablet_bootstrap.cc:492] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f: No bootstrap required, opened a new log
I20260926 16:48:52.154413 29470 ts_tablet_manager.cc:1397] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260926 16:48:52.154558 29472 tablet_bootstrap.cc:492] T 0f26235d3e464465b7adec3f462601c9 P 1a3714bdd55b428fb0654a39e5d385c0: No bootstrap required, opened a new log
I20260926 16:48:52.154556 29471 tablet_bootstrap.cc:492] T 0f26235d3e464465b7adec3f462601c9 P 2320aef7748c43ca8f2b9d0682a1c1aa: No bootstrap required, opened a new log
I20260926 16:48:52.154878 29472 ts_tablet_manager.cc:1397] T 0f26235d3e464465b7adec3f462601c9 P 1a3714bdd55b428fb0654a39e5d385c0: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260926 16:48:52.154878 29471 ts_tablet_manager.cc:1397] T 0f26235d3e464465b7adec3f462601c9 P 2320aef7748c43ca8f2b9d0682a1c1aa: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260926 16:48:52.163923 29470 raft_consensus.cc:348] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec66aae97d2a4bafb03391d535f27e6f" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 37305 } } peers { permanent_uuid: "2320aef7748c43ca8f2b9d0682a1c1aa" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46097 } } peers { permanent_uuid: "1a3714bdd55b428fb0654a39e5d385c0" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41197 } }
I20260926 16:48:52.164480 29470 raft_consensus.cc:374] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:48:52.164577 29470 raft_consensus.cc:729] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec66aae97d2a4bafb03391d535f27e6f, State: Initialized, Role: FOLLOWER
I20260926 16:48:52.165424 29470 consensus_queue.cc:260] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec66aae97d2a4bafb03391d535f27e6f" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 37305 } } peers { permanent_uuid: "2320aef7748c43ca8f2b9d0682a1c1aa" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46097 } } peers { permanent_uuid: "1a3714bdd55b428fb0654a39e5d385c0" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41197 } }
I20260926 16:48:52.166096 29472 raft_consensus.cc:348] T 0f26235d3e464465b7adec3f462601c9 P 1a3714bdd55b428fb0654a39e5d385c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec66aae97d2a4bafb03391d535f27e6f" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 37305 } } peers { permanent_uuid: "2320aef7748c43ca8f2b9d0682a1c1aa" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46097 } } peers { permanent_uuid: "1a3714bdd55b428fb0654a39e5d385c0" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41197 } }
I20260926 16:48:52.166095 29471 raft_consensus.cc:348] T 0f26235d3e464465b7adec3f462601c9 P 2320aef7748c43ca8f2b9d0682a1c1aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec66aae97d2a4bafb03391d535f27e6f" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 37305 } } peers { permanent_uuid: "2320aef7748c43ca8f2b9d0682a1c1aa" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46097 } } peers { permanent_uuid: "1a3714bdd55b428fb0654a39e5d385c0" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41197 } }
I20260926 16:48:52.166630 29471 raft_consensus.cc:374] T 0f26235d3e464465b7adec3f462601c9 P 2320aef7748c43ca8f2b9d0682a1c1aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:48:52.166630 29472 raft_consensus.cc:374] T 0f26235d3e464465b7adec3f462601c9 P 1a3714bdd55b428fb0654a39e5d385c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:48:52.166740 29471 raft_consensus.cc:729] T 0f26235d3e464465b7adec3f462601c9 P 2320aef7748c43ca8f2b9d0682a1c1aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2320aef7748c43ca8f2b9d0682a1c1aa, State: Initialized, Role: FOLLOWER
I20260926 16:48:52.166735 29472 raft_consensus.cc:729] T 0f26235d3e464465b7adec3f462601c9 P 1a3714bdd55b428fb0654a39e5d385c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a3714bdd55b428fb0654a39e5d385c0, State: Initialized, Role: FOLLOWER
I20260926 16:48:52.166878 29320 heartbeater.cc:503] Master 127.9.238.254:43317 was elected leader, sending a full tablet report...
I20260926 16:48:52.167541 29471 consensus_queue.cc:260] T 0f26235d3e464465b7adec3f462601c9 P 2320aef7748c43ca8f2b9d0682a1c1aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec66aae97d2a4bafb03391d535f27e6f" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 37305 } } peers { permanent_uuid: "2320aef7748c43ca8f2b9d0682a1c1aa" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46097 } } peers { permanent_uuid: "1a3714bdd55b428fb0654a39e5d385c0" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41197 } }
I20260926 16:48:52.167537 29472 consensus_queue.cc:260] T 0f26235d3e464465b7adec3f462601c9 P 1a3714bdd55b428fb0654a39e5d385c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec66aae97d2a4bafb03391d535f27e6f" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 37305 } } peers { permanent_uuid: "2320aef7748c43ca8f2b9d0682a1c1aa" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46097 } } peers { permanent_uuid: "1a3714bdd55b428fb0654a39e5d385c0" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41197 } }
I20260926 16:48:52.167776 29470 ts_tablet_manager.cc:1428] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260926 16:48:52.169587 29451 heartbeater.cc:503] Master 127.9.238.254:43317 was elected leader, sending a full tablet report...
I20260926 16:48:52.169808 29472 ts_tablet_manager.cc:1428] T 0f26235d3e464465b7adec3f462601c9 P 1a3714bdd55b428fb0654a39e5d385c0: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260926 16:48:52.170018 29471 ts_tablet_manager.cc:1428] T 0f26235d3e464465b7adec3f462601c9 P 2320aef7748c43ca8f2b9d0682a1c1aa: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
W20260926 16:48:52.193449 29321 tablet.cc:2369] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:48:52.272359 29476 raft_consensus.cc:482] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:48:52.272738 29476 raft_consensus.cc:504] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec66aae97d2a4bafb03391d535f27e6f" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 37305 } } peers { permanent_uuid: "2320aef7748c43ca8f2b9d0682a1c1aa" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46097 } } peers { permanent_uuid: "1a3714bdd55b428fb0654a39e5d385c0" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41197 } }
I20260926 16:48:52.274670 29476 leader_election.cc:288] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2320aef7748c43ca8f2b9d0682a1c1aa (127.9.238.195:46097), 1a3714bdd55b428fb0654a39e5d385c0 (127.9.238.193:41197)
W20260926 16:48:52.276816 29452 tablet.cc:2369] T 0f26235d3e464465b7adec3f462601c9 P 2320aef7748c43ca8f2b9d0682a1c1aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:48:52.283499 29467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:52.275144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:55727 (local address 127.9.238.195:46097)
0926 16:48:52.275397 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:52.275403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:52.275896 (+ 493us) server_negotiation.cc:407] Connection header received
0926 16:48:52.276223 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:52.276230 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:52.276316 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:52.276394 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:48:52.277658 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.278873 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.281446 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.282387 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.282578 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:52.282661 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:52.283033 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 16:48:52.283184 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260926 16:48:52.283644 29479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:52.274915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:46097 (local address 127.9.238.194:55727)
0926 16:48:52.275730 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:52.275760 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:52.276042 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:52.276620 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:52.276625 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:52.276659 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:48:52.277093 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:52.277105 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.279043 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.279049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:52.281178 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:52.281192 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.282627 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.282633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:52.282839 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:52.282883 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 16:48:52.282993 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 16:48:52.283184 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":51,"thread_start_us":366,"threads_started":1}
I20260926 16:48:52.284832 29469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:52.276090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:43893 (local address 127.9.238.193:41197)
0926 16:48:52.276285 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:52.276292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:52.276816 (+ 524us) server_negotiation.cc:407] Connection header received
0926 16:48:52.277370 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:52.277376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:52.277467 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:52.277539 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:48:52.278347 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.279405 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.282258 (+ 2853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.283434 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.283672 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:52.283743 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:52.284367 (+ 624us) server_negotiation.cc:299] Negotiation successful
0926 16:48:52.284561 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 16:48:52.284901 29480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:52.275815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:41197 (local address 127.9.238.194:43893)
0926 16:48:52.276663 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:52.276699 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:52.277173 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:52.277772 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:52.277775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:52.277797 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:48:52.278158 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:52.278170 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.279609 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.279615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:52.282032 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:52.282050 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.283904 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.283910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:52.284124 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:52.284162 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 16:48:52.284275 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 16:48:52.284469 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":1,"thread_start_us":477,"threads_started":1}
I20260926 16:48:52.284874 29406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f26235d3e464465b7adec3f462601c9" candidate_uuid: "ec66aae97d2a4bafb03391d535f27e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2320aef7748c43ca8f2b9d0682a1c1aa" is_pre_election: true
I20260926 16:48:52.285519 29406 raft_consensus.cc:2454] T 0f26235d3e464465b7adec3f462601c9 P 2320aef7748c43ca8f2b9d0682a1c1aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec66aae97d2a4bafb03391d535f27e6f in term 0.
I20260926 16:48:52.286279 29209 leader_election.cc:302] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2320aef7748c43ca8f2b9d0682a1c1aa, ec66aae97d2a4bafb03391d535f27e6f; no voters:
I20260926 16:48:52.286304 29144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f26235d3e464465b7adec3f462601c9" candidate_uuid: "ec66aae97d2a4bafb03391d535f27e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a3714bdd55b428fb0654a39e5d385c0" is_pre_election: true
I20260926 16:48:52.286713 29476 raft_consensus.cc:2781] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:48:52.286826 29476 raft_consensus.cc:482] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:48:52.286895 29476 raft_consensus.cc:3037] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:48:52.286971 29144 raft_consensus.cc:2454] T 0f26235d3e464465b7adec3f462601c9 P 1a3714bdd55b428fb0654a39e5d385c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec66aae97d2a4bafb03391d535f27e6f in term 0.
I20260926 16:48:52.289441 29476 raft_consensus.cc:504] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec66aae97d2a4bafb03391d535f27e6f" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 37305 } } peers { permanent_uuid: "2320aef7748c43ca8f2b9d0682a1c1aa" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46097 } } peers { permanent_uuid: "1a3714bdd55b428fb0654a39e5d385c0" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41197 } }
I20260926 16:48:52.290097 29476 leader_election.cc:288] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f [CANDIDATE]: Term 1 election: Requested vote from peers 2320aef7748c43ca8f2b9d0682a1c1aa (127.9.238.195:46097), 1a3714bdd55b428fb0654a39e5d385c0 (127.9.238.193:41197)
I20260926 16:48:52.290675 29406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f26235d3e464465b7adec3f462601c9" candidate_uuid: "ec66aae97d2a4bafb03391d535f27e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2320aef7748c43ca8f2b9d0682a1c1aa"
I20260926 16:48:52.290722 29144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f26235d3e464465b7adec3f462601c9" candidate_uuid: "ec66aae97d2a4bafb03391d535f27e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a3714bdd55b428fb0654a39e5d385c0"
I20260926 16:48:52.290910 29406 raft_consensus.cc:3037] T 0f26235d3e464465b7adec3f462601c9 P 2320aef7748c43ca8f2b9d0682a1c1aa [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:48:52.290957 29144 raft_consensus.cc:3037] T 0f26235d3e464465b7adec3f462601c9 P 1a3714bdd55b428fb0654a39e5d385c0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:48:52.294274 29144 raft_consensus.cc:2454] T 0f26235d3e464465b7adec3f462601c9 P 1a3714bdd55b428fb0654a39e5d385c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec66aae97d2a4bafb03391d535f27e6f in term 1.
I20260926 16:48:52.294274 29406 raft_consensus.cc:2454] T 0f26235d3e464465b7adec3f462601c9 P 2320aef7748c43ca8f2b9d0682a1c1aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec66aae97d2a4bafb03391d535f27e6f in term 1.
I20260926 16:48:52.294795 29209 leader_election.cc:302] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2320aef7748c43ca8f2b9d0682a1c1aa, ec66aae97d2a4bafb03391d535f27e6f; no voters:
I20260926 16:48:52.295210 29476 raft_consensus.cc:2781] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:48:52.295814 29476 raft_consensus.cc:686] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f [term 1 LEADER]: Becoming Leader. State: Replica: ec66aae97d2a4bafb03391d535f27e6f, State: Running, Role: LEADER
I20260926 16:48:52.296365 29476 consensus_queue.cc:237] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec66aae97d2a4bafb03391d535f27e6f" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 37305 } } peers { permanent_uuid: "2320aef7748c43ca8f2b9d0682a1c1aa" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46097 } } peers { permanent_uuid: "1a3714bdd55b428fb0654a39e5d385c0" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41197 } }
I20260926 16:48:52.302268 29000 catalog_manager.cc:5495] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f reported cstate change: term changed from 0 to 1, leader changed from <none> to ec66aae97d2a4bafb03391d535f27e6f (127.9.238.194). New cstate: current_term: 1 leader_uuid: "ec66aae97d2a4bafb03391d535f27e6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec66aae97d2a4bafb03391d535f27e6f" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 37305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2320aef7748c43ca8f2b9d0682a1c1aa" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a3714bdd55b428fb0654a39e5d385c0" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41197 } health_report { overall_health: UNKNOWN } } }
I20260926 16:48:52.332667 29465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:52.326809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40554 (local address 127.9.238.194:37305)
0926 16:48:52.327086 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:52.327092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:52.327109 (+ 17us) server_negotiation.cc:407] Connection header received
0926 16:48:52.327200 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:52.327204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:52.327298 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:52.327486 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:48:52.327895 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.328643 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.330092 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.330522 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.330659 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:52.331280 (+ 621us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:48:52.331995 (+ 715us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:48:52.332202 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:52.332331 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 16:48:52.332377 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260926 16:48:52.345139 29144 raft_consensus.cc:1261] T 0f26235d3e464465b7adec3f462601c9 P 1a3714bdd55b428fb0654a39e5d385c0 [term 1 FOLLOWER]: Refusing update from remote peer ec66aae97d2a4bafb03391d535f27e6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:48:52.345139 29406 raft_consensus.cc:1261] T 0f26235d3e464465b7adec3f462601c9 P 2320aef7748c43ca8f2b9d0682a1c1aa [term 1 FOLLOWER]: Refusing update from remote peer ec66aae97d2a4bafb03391d535f27e6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:48:52.346835 29476 consensus_queue.cc:1035] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2320aef7748c43ca8f2b9d0682a1c1aa" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 46097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:48:52.347258 29481 consensus_queue.cc:1035] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a3714bdd55b428fb0654a39e5d385c0" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:48:52.385030 29001 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:48:52.385502 29001 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:48:52.391182 29001 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee29643ab6c84ad9b47f1d9132db07d2: Sending DeleteTablet for 3 replicas of tablet 0f26235d3e464465b7adec3f462601c9
I20260926 16:48:52.392910 29255 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f26235d3e464465b7adec3f462601c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:48:52 UTC) from {username='slave'} at 127.0.0.1:40544
I20260926 16:48:52.393105 29124 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f26235d3e464465b7adec3f462601c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:48:52 UTC) from {username='slave'} at 127.0.0.1:49182
I20260926 16:48:52.393181 29386 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f26235d3e464465b7adec3f462601c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:48:52 UTC) from {username='slave'} at 127.0.0.1:53876
I20260926 16:48:52.397997 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 29062
I20260926 16:48:52.411068 29494 tablet_replica.cc:330] T 0f26235d3e464465b7adec3f462601c9 P 1a3714bdd55b428fb0654a39e5d385c0: stopping tablet replica
I20260926 16:48:52.411943 29494 raft_consensus.cc:2229] T 0f26235d3e464465b7adec3f462601c9 P 1a3714bdd55b428fb0654a39e5d385c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:48:52.415375 29494 raft_consensus.cc:2258] T 0f26235d3e464465b7adec3f462601c9 P 1a3714bdd55b428fb0654a39e5d385c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:48:52.417852 29496 tablet_replica.cc:330] T 0f26235d3e464465b7adec3f462601c9 P 2320aef7748c43ca8f2b9d0682a1c1aa: stopping tablet replica
I20260926 16:48:52.418329 29496 raft_consensus.cc:2229] T 0f26235d3e464465b7adec3f462601c9 P 2320aef7748c43ca8f2b9d0682a1c1aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:48:52.418838 29496 raft_consensus.cc:2258] T 0f26235d3e464465b7adec3f462601c9 P 2320aef7748c43ca8f2b9d0682a1c1aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:48:52.419492 29469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:52.399681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49196 (local address 127.9.238.193:41197)
0926 16:48:52.399937 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:52.399946 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:52.402798 (+ 2852us) server_negotiation.cc:407] Connection header received
0926 16:48:52.403726 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:52.403732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:52.403867 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:52.403948 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:52.405288 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.406792 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.412326 (+ 5534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.413273 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.413411 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:52.417390 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:52.417404 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:52.417424 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:48:52.418260 (+ 836us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:52.418330 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:48:52.418360 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:52.418457 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:52.418822 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 16:48:52.418998 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 16:48:52.419901 29494 ts_tablet_manager.cc:1905] T 0f26235d3e464465b7adec3f462601c9 P 1a3714bdd55b428fb0654a39e5d385c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:48:52.420948 29495 tablet_replica.cc:330] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f: stopping tablet replica
I20260926 16:48:52.421290 29184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:48:52.421579 29495 raft_consensus.cc:2229] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:48:52.422659 29495 raft_consensus.cc:2258] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:48:52.424749 29496 ts_tablet_manager.cc:1905] T 0f26235d3e464465b7adec3f462601c9 P 2320aef7748c43ca8f2b9d0682a1c1aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:48:52.425923 29495 ts_tablet_manager.cc:1905] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:48:52.435948 29495 ts_tablet_manager.cc:1918] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:48:52.436214 29495 log.cc:1196] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-1/wal/wals/0f26235d3e464465b7adec3f462601c9
I20260926 16:48:52.436959 29495 ts_tablet_manager.cc:1939] T 0f26235d3e464465b7adec3f462601c9 P ec66aae97d2a4bafb03391d535f27e6f: Deleting consensus metadata
I20260926 16:48:52.437964 29496 ts_tablet_manager.cc:1918] T 0f26235d3e464465b7adec3f462601c9 P 2320aef7748c43ca8f2b9d0682a1c1aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:48:52.438253 29496 log.cc:1196] T 0f26235d3e464465b7adec3f462601c9 P 2320aef7748c43ca8f2b9d0682a1c1aa: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790440988577176-10171-0/minicluster-data/ts-2/wal/wals/0f26235d3e464465b7adec3f462601c9
I20260926 16:48:52.439822 28988 catalog_manager.cc:4841] TS ec66aae97d2a4bafb03391d535f27e6f (127.9.238.194:37305): tablet 0f26235d3e464465b7adec3f462601c9 (table test-table [id=bdc598efeaf142f29111243528208446]) successfully deleted
I20260926 16:48:52.441015 29496 ts_tablet_manager.cc:1939] T 0f26235d3e464465b7adec3f462601c9 P 2320aef7748c43ca8f2b9d0682a1c1aa: Deleting consensus metadata
I20260926 16:48:52.444633 28988 catalog_manager.cc:4841] TS 2320aef7748c43ca8f2b9d0682a1c1aa (127.9.238.195:46097): tablet 0f26235d3e464465b7adec3f462601c9 (table test-table [id=bdc598efeaf142f29111243528208446]) successfully deleted
I20260926 16:48:52.566167 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 29062
I20260926 16:48:52.589215 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 29193
W20260926 16:48:52.589250 28988 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.238.193:41197 (error 108)
I20260926 16:48:52.592015 29464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:52.591188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:41197 (local address 127.0.0.1:49206)
0926 16:48:52.591617 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:52.591787 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:41197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199}
W20260926 16:48:52.592368 28988 catalog_manager.cc:4584] TS 1a3714bdd55b428fb0654a39e5d385c0 (127.9.238.193:41197): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f26235d3e464465b7adec3f462601c9: Network error: Client connection negotiation failed: client connection to 127.9.238.193:41197: connect: Connection refused (error 111)
I20260926 16:48:52.599161 29465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:52.590548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40564 (local address 127.9.238.194:37305)
0926 16:48:52.590763 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:52.590770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:52.590798 (+ 28us) server_negotiation.cc:407] Connection header received
0926 16:48:52.590985 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:52.590991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:52.591104 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:52.591240 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:52.591951 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.593241 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.594781 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.595327 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.595496 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:52.597101 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:52.597111 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:52.597137 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:48:52.598030 (+ 893us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:52.598097 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:48:52.598123 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:52.598274 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:52.598522 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 16:48:52.598666 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 16:48:52.600319 29315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:48:52.646603 29464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:52.646087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:41197 (local address 127.0.0.1:49214)
0926 16:48:52.646326 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:52.646417 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:41197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 16:48:52.647763 29464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:52.647337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:41197 (local address 127.0.0.1:49216)
0926 16:48:52.647501 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:52.647634 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:41197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 16:48:52.708253 29464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:52.707800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:41197 (local address 127.0.0.1:49220)
0926 16:48:52.707990 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:52.708104 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:41197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 16:48:52.709290 29464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:52.708930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:41197 (local address 127.0.0.1:49236)
0926 16:48:52.709090 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:52.709155 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:41197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 16:48:52.734972 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 29193
I20260926 16:48:52.758631 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 29324
I20260926 16:48:52.766178 29467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:52.759526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53884 (local address 127.9.238.195:46097)
0926 16:48:52.759777 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:52.759781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:52.759909 (+ 128us) server_negotiation.cc:407] Connection header received
0926 16:48:52.760259 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:52.760264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:52.760367 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:52.760484 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:52.760888 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.761675 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.762973 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:52.763349 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:52.763445 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:52.764772 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:52.764780 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:52.764796 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:48:52.765446 (+ 650us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:52.765497 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:48:52.765517 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:52.765615 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:52.765754 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 16:48:52.765854 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260926 16:48:52.766876 29446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:48:52.789844 29464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:52.789337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:41197 (local address 127.0.0.1:49252)
0926 16:48:52.789593 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:52.789697 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:41197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 16:48:52.791442 29464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:52.790961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:41197 (local address 127.0.0.1:49266)
0926 16:48:52.791205 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:52.791296 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:41197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260926 16:48:52.867527 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 29324
I20260926 16:48:52.886322 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 28972
I20260926 16:48:52.887972 29033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:48:53.020819 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 28972
2026-09-26T16:48:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3329 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:48:53 dist-test-slave-wt2j krb5kdc[29505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:48:53 dist-test-slave-wt2j krb5kdc[29505](info): set up 2 sockets
Sep 26 16:48:53 dist-test-slave-wt2j krb5kdc[29505](info): commencing operation
krb5kdc: starting...
W20260926 16:48:54.201532 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:48:54 dist-test-slave-wt2j krb5kdc[29505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:48:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:48:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:48:54.513425 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:44489
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:45601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:44489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:48:54.880185 29518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:48:54.880549 29518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:48:54.880728 29518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:48:54.888180 29518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:48:54.888340 29518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:48:54.888432 29518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:48:54.888541 29518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:48:54.888629 29518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:48:54.898803 29518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:45601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:44489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:44489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:48:54.900357 29518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:48:54.901727 29518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:48:54.915524 29523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:54.915644 29526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:54.915699 29524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:48:54.916543 29518 server_base.cc:1034] running on GCE node
I20260926 16:48:54.917584 29518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:48:54.919529 29518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:48:54.920861 29518 hybrid_clock.cc:648] HybridClock initialized: now 1790441334920790 us; error 45 us; skew 500 ppm
I20260926 16:48:54.921307 29518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:48:54 dist-test-slave-wt2j krb5kdc[29505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441334, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:48:54.932101 29518 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:48:54.934407 29518 webserver.cc:460] Webserver started at http://127.9.238.254:46385/ using document root <none> and password file <none>
I20260926 16:48:54.935412 29518 fs_manager.cc:359] Metadata directory not provided
I20260926 16:48:54.935626 29518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:48:54.936043 29518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:48:54.940169 29518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "faf29b2c8f404663823231e6bfc27649"
format_stamp: "Formatted at 2026-09-26 16:48:54 on dist-test-slave-wt2j"
server_key: "519acc44c3a4cb94b04fcaaa58d19f83"
server_key_iv: "1e5565e3792bf6371090e92629bdbf14"
server_key_version: "encryptionkey@0"
I20260926 16:48:54.941145 29518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "faf29b2c8f404663823231e6bfc27649"
format_stamp: "Formatted at 2026-09-26 16:48:54 on dist-test-slave-wt2j"
server_key: "519acc44c3a4cb94b04fcaaa58d19f83"
server_key_iv: "1e5565e3792bf6371090e92629bdbf14"
server_key_version: "encryptionkey@0"
I20260926 16:48:54.948141 29518 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.006s
I20260926 16:48:54.952865 29533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:54.954742 29518 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 16:48:54.955031 29518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "faf29b2c8f404663823231e6bfc27649"
format_stamp: "Formatted at 2026-09-26 16:48:54 on dist-test-slave-wt2j"
server_key: "519acc44c3a4cb94b04fcaaa58d19f83"
server_key_iv: "1e5565e3792bf6371090e92629bdbf14"
server_key_version: "encryptionkey@0"
I20260926 16:48:54.955302 29518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:48:54.994552 29518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:48:55.000385 29518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:48:55.000792 29518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:48:55.026172 29518 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:44489
I20260926 16:48:55.026216 29584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:44489 every 8 connection(s)
I20260926 16:48:55.027987 29518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:48:55.032444 29585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:48:55.036681 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 29518
I20260926 16:48:55.037022 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:48:55.037600 10171 external_mini_cluster.cc:1428] Setting key 7bb0e66ee98ee1be9a65e08072fbb5a9
I20260926 16:48:55.044600 29585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faf29b2c8f404663823231e6bfc27649: Bootstrap starting.
I20260926 16:48:55.048679 29585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P faf29b2c8f404663823231e6bfc27649: Neither blocks nor log segments found. Creating new log.
I20260926 16:48:55.049985 29585 log.cc:824] T 00000000000000000000000000000000 P faf29b2c8f404663823231e6bfc27649: Log is configured to *not* fsync() on all Append() calls
I20260926 16:48:55.054234 29585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faf29b2c8f404663823231e6bfc27649: No bootstrap required, opened a new log
Sep 26 16:48:55 dist-test-slave-wt2j krb5kdc[29505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:48:55.063156 29585 raft_consensus.cc:348] T 00000000000000000000000000000000 P faf29b2c8f404663823231e6bfc27649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf29b2c8f404663823231e6bfc27649" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44489 } }
I20260926 16:48:55.063632 29585 raft_consensus.cc:374] T 00000000000000000000000000000000 P faf29b2c8f404663823231e6bfc27649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:48:55.063740 29585 raft_consensus.cc:729] T 00000000000000000000000000000000 P faf29b2c8f404663823231e6bfc27649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faf29b2c8f404663823231e6bfc27649, State: Initialized, Role: FOLLOWER
I20260926 16:48:55.064473 29585 consensus_queue.cc:260] T 00000000000000000000000000000000 P faf29b2c8f404663823231e6bfc27649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf29b2c8f404663823231e6bfc27649" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44489 } }
I20260926 16:48:55.064741 29585 raft_consensus.cc:388] T 00000000000000000000000000000000 P faf29b2c8f404663823231e6bfc27649 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:48:55.064893 29585 raft_consensus.cc:482] T 00000000000000000000000000000000 P faf29b2c8f404663823231e6bfc27649 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:48:55.065109 29585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P faf29b2c8f404663823231e6bfc27649 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:48:55.068576 29585 raft_consensus.cc:504] T 00000000000000000000000000000000 P faf29b2c8f404663823231e6bfc27649 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf29b2c8f404663823231e6bfc27649" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44489 } }
I20260926 16:48:55.068867 29588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:55.040261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53802 (local address 127.9.238.254:44489)
0926 16:48:55.041468 (+ 1207us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:55.041483 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:55.041543 (+ 60us) server_negotiation.cc:407] Connection header received
0926 16:48:55.044034 (+ 2491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:55.044069 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:55.044637 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:55.045190 (+ 553us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:55.047212 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:55.049094 (+ 1882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:55.051128 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:55.052338 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:55.052550 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:55.058335 (+ 5785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:55.058345 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:55.058368 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:48:55.058416 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:55.062942 (+ 4526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:55.064146 (+ 1204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:55.064161 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:55.064169 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:55.064384 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:55.065002 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:55.065010 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:55.065014 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:55.065959 (+ 945us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:55.066349 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:55.066847 (+ 498us) server_negotiation.cc:299] Negotiation successful
0926 16:48:55.067405 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":891,"thread_start_us":577,"threads_started":1}
I20260926 16:48:55.069204 29585 leader_election.cc:302] T 00000000000000000000000000000000 P faf29b2c8f404663823231e6bfc27649 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: faf29b2c8f404663823231e6bfc27649; no voters:
I20260926 16:48:55.069984 29585 leader_election.cc:288] T 00000000000000000000000000000000 P faf29b2c8f404663823231e6bfc27649 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:48:55.070207 29590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P faf29b2c8f404663823231e6bfc27649 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:48:55.071954 29590 raft_consensus.cc:686] T 00000000000000000000000000000000 P faf29b2c8f404663823231e6bfc27649 [term 1 LEADER]: Becoming Leader. State: Replica: faf29b2c8f404663823231e6bfc27649, State: Running, Role: LEADER
I20260926 16:48:55.072554 29585 sys_catalog.cc:564] T 00000000000000000000000000000000 P faf29b2c8f404663823231e6bfc27649 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:48:55.072840 29590 consensus_queue.cc:237] T 00000000000000000000000000000000 P faf29b2c8f404663823231e6bfc27649 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf29b2c8f404663823231e6bfc27649" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44489 } }
I20260926 16:48:55.077221 29591 sys_catalog.cc:455] T 00000000000000000000000000000000 P faf29b2c8f404663823231e6bfc27649 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "faf29b2c8f404663823231e6bfc27649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf29b2c8f404663823231e6bfc27649" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44489 } } }
I20260926 16:48:55.077656 29591 sys_catalog.cc:458] T 00000000000000000000000000000000 P faf29b2c8f404663823231e6bfc27649 [sys.catalog]: This master's current role is: LEADER
I20260926 16:48:55.078346 29592 sys_catalog.cc:455] T 00000000000000000000000000000000 P faf29b2c8f404663823231e6bfc27649 [sys.catalog]: SysCatalogTable state changed. Reason: New leader faf29b2c8f404663823231e6bfc27649. Latest consensus state: current_term: 1 leader_uuid: "faf29b2c8f404663823231e6bfc27649" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf29b2c8f404663823231e6bfc27649" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44489 } } }
I20260926 16:48:55.078691 29592 sys_catalog.cc:458] T 00000000000000000000000000000000 P faf29b2c8f404663823231e6bfc27649 [sys.catalog]: This master's current role is: LEADER
I20260926 16:48:55.078774 29598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:48:55.093102 29598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:48:55.106283 29598 catalog_manager.cc:1347] Generated new cluster ID: c665326f31c14f91986a6af18b9f6ca2
I20260926 16:48:55.106458 29598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:48:55.131973 29598 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:48:55.133648 29598 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:48:55.150049 29598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P faf29b2c8f404663823231e6bfc27649: Generated new TSK 0
I20260926 16:48:55.151620 29598 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.9.238.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
Entry for principal kudu/127.9.238.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab.
I20260926 16:48:55.220790 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:0
--local_ip_for_outbound_sockets=127.9.238.193
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44489
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:45601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:48:55.564327 29611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:48:55.564644 29611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:48:55.564796 29611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:48:55.572563 29611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:48:55.572700 29611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:48:55.572903 29611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:48:55.585074 29611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:45601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.193.keytab
--principal=kudu/127.9.238.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:48:55.586444 29611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:48:55.587906 29611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:48:55.602869 29619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:55.602931 29617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:55.602905 29616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:48:55.603940 29611 server_base.cc:1034] running on GCE node
I20260926 16:48:55.604668 29611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:48:55.605809 29611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:48:55.607084 29611 hybrid_clock.cc:648] HybridClock initialized: now 1790441335607032 us; error 49 us; skew 500 ppm
I20260926 16:48:55.607450 29611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:48:55 dist-test-slave-wt2j krb5kdc[29505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441335, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:48:55.611228 29611 init.cc:377] Logged in from keytab as kudu/127.9.238.193@KRBTEST.COM (short username kudu)
I20260926 16:48:55.613179 29611 webserver.cc:460] Webserver started at http://127.9.238.193:36551/ using document root <none> and password file <none>
I20260926 16:48:55.613930 29611 fs_manager.cc:359] Metadata directory not provided
I20260926 16:48:55.614089 29611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:48:55.614419 29611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:48:55.617522 29611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-0/data/instance:
uuid: "998ecc92cab640f991664783e6c0d19a"
format_stamp: "Formatted at 2026-09-26 16:48:55 on dist-test-slave-wt2j"
server_key: "b8d5c4edd683b6c4359ea457c1b28787"
server_key_iv: "159ebaa6df359d7e1df5119af59736e0"
server_key_version: "encryptionkey@0"
I20260926 16:48:55.618317 29611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance:
uuid: "998ecc92cab640f991664783e6c0d19a"
format_stamp: "Formatted at 2026-09-26 16:48:55 on dist-test-slave-wt2j"
server_key: "b8d5c4edd683b6c4359ea457c1b28787"
server_key_iv: "159ebaa6df359d7e1df5119af59736e0"
server_key_version: "encryptionkey@0"
I20260926 16:48:55.624379 29611 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260926 16:48:55.628614 29626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:55.630321 29611 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:48:55.630550 29611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "998ecc92cab640f991664783e6c0d19a"
format_stamp: "Formatted at 2026-09-26 16:48:55 on dist-test-slave-wt2j"
server_key: "b8d5c4edd683b6c4359ea457c1b28787"
server_key_iv: "159ebaa6df359d7e1df5119af59736e0"
server_key_version: "encryptionkey@0"
I20260926 16:48:55.630790 29611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:48:55.654127 29611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:48:55.659099 29611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:48:55.659412 29611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:48:55.660668 29611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:48:55.662215 29611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:48:55.662325 29611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:55.662526 29611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:48:55.662600 29611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:55.703222 29611 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:41211
I20260926 16:48:55.703258 29738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:41211 every 8 connection(s)
I20260926 16:48:55.704653 29611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:48:55.707764 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 29611
I20260926 16:48:55.707984 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance
I20260926 16:48:55.708673 10171 external_mini_cluster.cc:1428] Setting key 92ffeec7fca99cee1fb48e7deb98adad
Sep 26 16:48:55 dist-test-slave-wt2j krb5kdc[29505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441335, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.193@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:48:55.735602 29741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:55.708466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:41539 (local address 127.9.238.254:44489)
0926 16:48:55.709141 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:55.709147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:55.710941 (+ 1794us) server_negotiation.cc:407] Connection header received
0926 16:48:55.712743 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:55.712753 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:55.712881 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:55.713049 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:55.714672 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:55.716054 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:55.718401 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:55.719115 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:55.719262 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:55.725426 (+ 6164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:55.725436 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:55.725441 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:48:55.725485 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:55.729528 (+ 4043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:55.730583 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:55.730587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:55.730589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:55.730705 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:55.731329 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:55.731334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:55.731338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:55.731774 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:55.731917 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:55.734850 (+ 2933us) server_negotiation.cc:299] Negotiation successful
0926 16:48:55.735141 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":376,"threads_started":1}
I20260926 16:48:55.736631 29742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:55.709106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:44489 (local address 127.9.238.193:41539)
0926 16:48:55.710136 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:55.710194 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:55.712388 (+ 2194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:55.713301 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:55.713309 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:55.713860 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:48:55.714453 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:55.714471 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:55.716292 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:55.716298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:55.718190 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:55.718206 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:55.719684 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:55.719693 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:55.720077 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:55.720945 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:48:55.720986 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:48:55.725134 (+ 4148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:48:55.729680 (+ 4546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:48:55.729690 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:48:55.729709 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:48:55.730418 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:48:55.730867 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:48:55.730874 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:48:55.730880 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:48:55.731187 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:48:55.732009 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:48:55.732021 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:48:55.732539 (+ 518us) client_negotiation.cc:769] Sending connection context
0926 16:48:55.732898 (+ 359us) client_negotiation.cc:240] Negotiation successful
0926 16:48:55.733554 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":457,"threads_started":1}
I20260926 16:48:55.739233 29739 heartbeater.cc:347] Connected to a master server at 127.9.238.254:44489
I20260926 16:48:55.739667 29739 heartbeater.cc:464] Registering TS with master...
I20260926 16:48:55.740613 29739 heartbeater.cc:511] Master 127.9.238.254:44489 requested a full tablet report, sending...
I20260926 16:48:55.743275 29550 ts_manager.cc:194] Registered new tserver with Master: 998ecc92cab640f991664783e6c0d19a (127.9.238.193:41211)
I20260926 16:48:55.745216 29550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.193@KRBTEST.COM'} at 127.9.238.193:41539
WARNING: no policy specified for kudu/127.9.238.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
Entry for principal kudu/127.9.238.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab.
I20260926 16:48:55.799965 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:0
--local_ip_for_outbound_sockets=127.9.238.194
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44489
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:45601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:48:56.178020 29745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:48:56.178329 29745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:48:56.178467 29745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:48:56.186187 29745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:48:56.186322 29745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:48:56.186525 29745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:48:56.197041 29745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:45601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.194.keytab
--principal=kudu/127.9.238.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:48:56.198457 29745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:48:56.199944 29745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:48:56.215703 29751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:56.215761 29753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:56.215778 29750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:48:56.216917 29745 server_base.cc:1034] running on GCE node
I20260926 16:48:56.217759 29745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:48:56.219121 29745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:48:56.220445 29745 hybrid_clock.cc:648] HybridClock initialized: now 1790441336220395 us; error 42 us; skew 500 ppm
I20260926 16:48:56.220855 29745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:48:56 dist-test-slave-wt2j krb5kdc[29505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441336, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:48:56.224973 29745 init.cc:377] Logged in from keytab as kudu/127.9.238.194@KRBTEST.COM (short username kudu)
I20260926 16:48:56.227094 29745 webserver.cc:460] Webserver started at http://127.9.238.194:34227/ using document root <none> and password file <none>
I20260926 16:48:56.228189 29745 fs_manager.cc:359] Metadata directory not provided
I20260926 16:48:56.228374 29745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:48:56.228780 29745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:48:56.232569 29745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-1/data/instance:
uuid: "a7ec24f381ef48ff9a05eaa54af4e21c"
format_stamp: "Formatted at 2026-09-26 16:48:56 on dist-test-slave-wt2j"
server_key: "3659618fa7a373260c15fe4a87aa2bfc"
server_key_iv: "b5733b238cc0413aa1c444174026c4d2"
server_key_version: "encryptionkey@0"
I20260926 16:48:56.233517 29745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance:
uuid: "a7ec24f381ef48ff9a05eaa54af4e21c"
format_stamp: "Formatted at 2026-09-26 16:48:56 on dist-test-slave-wt2j"
server_key: "3659618fa7a373260c15fe4a87aa2bfc"
server_key_iv: "b5733b238cc0413aa1c444174026c4d2"
server_key_version: "encryptionkey@0"
I20260926 16:48:56.241148 29745 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260926 16:48:56.246402 29760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:56.248677 29745 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260926 16:48:56.248982 29745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "a7ec24f381ef48ff9a05eaa54af4e21c"
format_stamp: "Formatted at 2026-09-26 16:48:56 on dist-test-slave-wt2j"
server_key: "3659618fa7a373260c15fe4a87aa2bfc"
server_key_iv: "b5733b238cc0413aa1c444174026c4d2"
server_key_version: "encryptionkey@0"
I20260926 16:48:56.249291 29745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:48:56.270558 29745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:48:56.276474 29745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:48:56.276837 29745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:48:56.278162 29745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:48:56.280011 29745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:48:56.280139 29745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:56.280339 29745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:48:56.280414 29745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:56.322731 29745 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:44071
I20260926 16:48:56.322780 29872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:44071 every 8 connection(s)
I20260926 16:48:56.324478 29745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:48:56.333809 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 29745
I20260926 16:48:56.334066 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance
I20260926 16:48:56.334635 10171 external_mini_cluster.cc:1428] Setting key 1c734ba58d89590c263fd460ad8001d6
Sep 26 16:48:56 dist-test-slave-wt2j krb5kdc[29505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441336, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.194@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:48:56.347582 29875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:56.327945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:57521 (local address 127.9.238.254:44489)
0926 16:48:56.328479 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:56.328484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:56.329793 (+ 1309us) server_negotiation.cc:407] Connection header received
0926 16:48:56.331872 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:56.331878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:56.331982 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:56.332120 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:56.333860 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:56.334772 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:56.336708 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:56.337366 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:56.337517 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:56.341348 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:56.341353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:56.341356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:48:56.341389 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:56.344274 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:56.345054 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:56.345060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:56.345065 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:56.345181 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:56.345627 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:56.345633 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:56.345636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:56.345931 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:56.346202 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:56.346920 (+ 718us) server_negotiation.cc:299] Negotiation successful
0926 16:48:56.347159 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":254,"threads_started":1}
I20260926 16:48:56.348582 29876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:56.328683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:44489 (local address 127.9.238.194:57521)
0926 16:48:56.329600 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:56.329653 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:56.331481 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:56.332381 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:56.332392 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:56.333006 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:48:56.333664 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:56.333682 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:56.334923 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:56.334935 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:56.336556 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:56.336574 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:56.337623 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:56.337630 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:56.337864 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:56.338687 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:48:56.338715 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:48:56.341101 (+ 2386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:48:56.344427 (+ 3326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:48:56.344437 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:48:56.344455 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:48:56.344911 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:48:56.345302 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:48:56.345307 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:48:56.345311 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:48:56.345498 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:48:56.346172 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:48:56.346180 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:48:56.346559 (+ 379us) client_negotiation.cc:769] Sending connection context
0926 16:48:56.346833 (+ 274us) client_negotiation.cc:240] Negotiation successful
0926 16:48:56.347381 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":347,"threads_started":1}
I20260926 16:48:56.350242 29873 heartbeater.cc:347] Connected to a master server at 127.9.238.254:44489
I20260926 16:48:56.350585 29873 heartbeater.cc:464] Registering TS with master...
I20260926 16:48:56.351351 29873 heartbeater.cc:511] Master 127.9.238.254:44489 requested a full tablet report, sending...
I20260926 16:48:56.353116 29550 ts_manager.cc:194] Registered new tserver with Master: a7ec24f381ef48ff9a05eaa54af4e21c (127.9.238.194:44071)
I20260926 16:48:56.354557 29550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.194@KRBTEST.COM'} at 127.9.238.194:57521
WARNING: no policy specified for kudu/127.9.238.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
Entry for principal kudu/127.9.238.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab.
I20260926 16:48:56.407270 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:0
--local_ip_for_outbound_sockets=127.9.238.195
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44489
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:45601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:48:56.749323 29739 heartbeater.cc:503] Master 127.9.238.254:44489 was elected leader, sending a full tablet report...
W20260926 16:48:56.764572 29879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:48:56.764922 29879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:48:56.765100 29879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:48:56.772883 29879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:48:56.773046 29879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:48:56.773279 29879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:48:56.784127 29879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:45601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.195.keytab
--principal=kudu/127.9.238.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:48:56.785612 29879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:48:56.786978 29879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:48:56.805294 29884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:56.805646 29887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:56.805912 29885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:48:56.806829 29879 server_base.cc:1034] running on GCE node
I20260926 16:48:56.807682 29879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:48:56.808983 29879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:48:56.810261 29879 hybrid_clock.cc:648] HybridClock initialized: now 1790441336810197 us; error 65 us; skew 500 ppm
I20260926 16:48:56.810670 29879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:48:56 dist-test-slave-wt2j krb5kdc[29505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441336, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:48:56.814550 29879 init.cc:377] Logged in from keytab as kudu/127.9.238.195@KRBTEST.COM (short username kudu)
I20260926 16:48:56.816402 29879 webserver.cc:460] Webserver started at http://127.9.238.195:42775/ using document root <none> and password file <none>
I20260926 16:48:56.817167 29879 fs_manager.cc:359] Metadata directory not provided
I20260926 16:48:56.817286 29879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:48:56.817564 29879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:48:56.820706 29879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-2/data/instance:
uuid: "0fc3233e51734688be72c68053f1415d"
format_stamp: "Formatted at 2026-09-26 16:48:56 on dist-test-slave-wt2j"
server_key: "89fff6b6fd65b9b19e2f1444c91aac3f"
server_key_iv: "59cc41ae3d488d41f31ed1cab2b7164d"
server_key_version: "encryptionkey@0"
I20260926 16:48:56.821540 29879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance:
uuid: "0fc3233e51734688be72c68053f1415d"
format_stamp: "Formatted at 2026-09-26 16:48:56 on dist-test-slave-wt2j"
server_key: "89fff6b6fd65b9b19e2f1444c91aac3f"
server_key_iv: "59cc41ae3d488d41f31ed1cab2b7164d"
server_key_version: "encryptionkey@0"
I20260926 16:48:56.827628 29879 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260926 16:48:56.832044 29894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:56.833823 29879 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260926 16:48:56.834043 29879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "0fc3233e51734688be72c68053f1415d"
format_stamp: "Formatted at 2026-09-26 16:48:56 on dist-test-slave-wt2j"
server_key: "89fff6b6fd65b9b19e2f1444c91aac3f"
server_key_iv: "59cc41ae3d488d41f31ed1cab2b7164d"
server_key_version: "encryptionkey@0"
I20260926 16:48:56.834267 29879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:48:56.850953 29879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:48:56.855531 29879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:48:56.855903 29879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:48:56.857028 29879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:48:56.858563 29879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:48:56.858671 29879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:56.858831 29879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:48:56.858923 29879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:56.903477 29879 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:40795
I20260926 16:48:56.903637 30006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:40795 every 8 connection(s)
I20260926 16:48:56.905004 29879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:48:56.911967 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 29879
I20260926 16:48:56.912173 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance
I20260926 16:48:56.912678 10171 external_mini_cluster.cc:1428] Setting key a3d5dc9cd74f939bb4053e6ee3308615
Sep 26 16:48:56 dist-test-slave-wt2j krb5kdc[29505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441336, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.195@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:48:56.932984 30009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:56.908649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:58109 (local address 127.9.238.254:44489)
0926 16:48:56.909245 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:56.909252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:56.910647 (+ 1395us) server_negotiation.cc:407] Connection header received
0926 16:48:56.913395 (+ 2748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:56.913401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:56.913562 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:56.913777 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:56.916176 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:56.917187 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:56.919192 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:56.919735 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:56.919855 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:56.924633 (+ 4778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:56.924641 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:56.924646 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:48:56.924700 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:56.928561 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:56.929757 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:56.929764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:56.929778 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:56.930010 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:56.930618 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:56.930623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:56.930626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:56.931023 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:56.931166 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:56.932342 (+ 1176us) server_negotiation.cc:299] Negotiation successful
0926 16:48:56.932603 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":264,"threads_started":1}
I20260926 16:48:56.933769 30010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:56.909323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:44489 (local address 127.9.238.195:58109)
0926 16:48:56.910395 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:56.910460 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:56.912924 (+ 2464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:56.914128 (+ 1204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:56.914138 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:56.914830 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:48:56.915902 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:56.915935 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:56.917401 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:56.917408 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:56.918972 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:56.918987 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:56.920016 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:56.920022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:56.920276 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:56.921122 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:48:56.921151 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:48:56.924270 (+ 3119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:48:56.928775 (+ 4505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:48:56.928790 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:48:56.928813 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:48:56.929533 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:48:56.930207 (+ 674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:48:56.930212 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:48:56.930216 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:48:56.930467 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:48:56.931350 (+ 883us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:48:56.931358 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:48:56.931874 (+ 516us) client_negotiation.cc:769] Sending connection context
0926 16:48:56.932148 (+ 274us) client_negotiation.cc:240] Negotiation successful
0926 16:48:56.932612 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":399,"threads_started":1}
I20260926 16:48:56.935621 30007 heartbeater.cc:347] Connected to a master server at 127.9.238.254:44489
I20260926 16:48:56.936105 30007 heartbeater.cc:464] Registering TS with master...
I20260926 16:48:56.937194 30007 heartbeater.cc:511] Master 127.9.238.254:44489 requested a full tablet report, sending...
I20260926 16:48:56.939646 29550 ts_manager.cc:194] Registered new tserver with Master: 0fc3233e51734688be72c68053f1415d (127.9.238.195:40795)
I20260926 16:48:56.940727 29550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.9.238.195@KRBTEST.COM'} at 127.9.238.195:58109
I20260926 16:48:56.943421 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:48:56.963135 30009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:56.948130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53816 (local address 127.9.238.254:44489)
0926 16:48:56.948554 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:56.948564 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:56.948963 (+ 399us) server_negotiation.cc:407] Connection header received
0926 16:48:56.949230 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:56.949235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:56.949316 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:56.949437 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:56.951061 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:56.951945 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:56.953540 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:56.954097 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:56.954403 (+ 306us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:56.957349 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:56.957355 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:56.957359 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:48:56.957397 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:56.960189 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:56.960949 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:56.960955 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:56.960959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:56.961085 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:56.961474 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:56.961481 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:56.961484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:56.961894 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:56.962163 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:56.962530 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 16:48:56.962796 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260926 16:48:56.967810 29550 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:48:56.974995 29550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:48:57.017963 30023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:56.993413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39420 (local address 127.9.238.194:44071)
0926 16:48:57.004348 (+ 10935us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:57.004364 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:57.007074 (+ 2710us) server_negotiation.cc:407] Connection header received
0926 16:48:57.007223 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:57.007233 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:57.007665 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:57.007906 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:48:57.009048 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:57.010692 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:57.014651 (+ 3959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:57.016424 (+ 1773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:57.016597 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:57.016888 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:57.017314 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 16:48:57.017546 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10630,"thread_start_us":10446,"threads_started":1}
I20260926 16:48:57.018077 30021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:56.992875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:44071 (local address 127.0.0.1:39420)
0926 16:48:57.006772 (+ 13897us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:57.006792 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:57.006927 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:57.008244 (+ 1317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:57.008252 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:57.008285 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:48:57.008826 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:57.008843 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:57.010881 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:57.010889 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:57.014413 (+ 3524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:57.014428 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:57.016673 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:57.016679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:57.016982 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:57.017056 (+ 74us) client_negotiation.cc:769] Sending connection context
0926 16:48:57.017233 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 16:48:57.017558 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13776,"mutex_wait_us":62,"thread_start_us":13641,"threads_started":1}
I20260926 16:48:57.020213 30020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:56.992603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:41211 (local address 127.0.0.1:59170)
0926 16:48:57.005546 (+ 12943us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:57.005632 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:57.006017 (+ 385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:57.008952 (+ 2935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:57.008957 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:57.008984 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:48:57.011895 (+ 2911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:57.011922 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:57.014170 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:57.014176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:57.016000 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:57.016015 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:57.019371 (+ 3356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:57.019380 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:57.019606 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:57.019642 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 16:48:57.019738 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 16:48:57.019880 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12382,"thread_start_us":12110,"threads_started":1}
I20260926 16:48:57.022055 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:57.006564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50844 (local address 127.9.238.195:40795)
0926 16:48:57.011657 (+ 5093us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:57.011667 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:57.012757 (+ 1090us) server_negotiation.cc:407] Connection header received
0926 16:48:57.013022 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:57.013029 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:57.013306 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:57.013528 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:48:57.014842 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:57.016624 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:57.019475 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:57.020902 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:57.021112 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:57.021336 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:57.021646 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 16:48:57.021791 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4838,"thread_start_us":4586,"threads_started":1}
I20260926 16:48:57.022171 30025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:57.008193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:40795 (local address 127.0.0.1:50844)
0926 16:48:57.012613 (+ 4420us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:57.012629 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:57.012808 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:57.013831 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:57.013836 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:57.013869 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:48:57.014585 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:57.014598 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:57.016829 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:57.016835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:57.019254 (+ 2419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:57.019269 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:57.021217 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:57.021223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:57.021416 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:57.021455 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 16:48:57.021566 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 16:48:57.021727 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4287,"mutex_wait_us":73,"thread_start_us":3516,"threads_started":1}
I20260926 16:48:57.023505 29808 tablet_service.cc:1467] Processing CreateTablet for tablet ed8519ae4bc6419ca2622f0477f66e15 (DEFAULT_TABLE table=test-table [id=8297bee3b362417083a35cf35e63d0ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:48:57.025274 29808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed8519ae4bc6419ca2622f0477f66e15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:48:57.028057 29942 tablet_service.cc:1467] Processing CreateTablet for tablet ed8519ae4bc6419ca2622f0477f66e15 (DEFAULT_TABLE table=test-table [id=8297bee3b362417083a35cf35e63d0ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:48:57.029744 29942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed8519ae4bc6419ca2622f0477f66e15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:48:57.035043 30022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:56.992894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59170 (local address 127.9.238.193:41211)
0926 16:48:57.008033 (+ 15139us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:57.008044 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:57.008092 (+ 48us) server_negotiation.cc:407] Connection header received
0926 16:48:57.008208 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:57.008216 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:57.008456 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:57.008671 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:48:57.012223 (+ 3552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:57.014007 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:57.016214 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:57.017822 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:57.018971 (+ 1149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:57.019218 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:57.034287 (+ 15069us) server_negotiation.cc:299] Negotiation successful
0926 16:48:57.034611 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14923,"thread_start_us":14684,"threads_started":1}
I20260926 16:48:57.040141 29674 tablet_service.cc:1467] Processing CreateTablet for tablet ed8519ae4bc6419ca2622f0477f66e15 (DEFAULT_TABLE table=test-table [id=8297bee3b362417083a35cf35e63d0ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:48:57.041760 29674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed8519ae4bc6419ca2622f0477f66e15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:48:57.052634 30026 tablet_bootstrap.cc:492] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c: Bootstrap starting.
I20260926 16:48:57.055944 30027 tablet_bootstrap.cc:492] T ed8519ae4bc6419ca2622f0477f66e15 P 0fc3233e51734688be72c68053f1415d: Bootstrap starting.
I20260926 16:48:57.057091 30026 tablet_bootstrap.cc:654] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c: Neither blocks nor log segments found. Creating new log.
I20260926 16:48:57.058023 30028 tablet_bootstrap.cc:492] T ed8519ae4bc6419ca2622f0477f66e15 P 998ecc92cab640f991664783e6c0d19a: Bootstrap starting.
I20260926 16:48:57.058974 30026 log.cc:824] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c: Log is configured to *not* fsync() on all Append() calls
I20260926 16:48:57.061972 30028 tablet_bootstrap.cc:654] T ed8519ae4bc6419ca2622f0477f66e15 P 998ecc92cab640f991664783e6c0d19a: Neither blocks nor log segments found. Creating new log.
I20260926 16:48:57.061971 30027 tablet_bootstrap.cc:654] T ed8519ae4bc6419ca2622f0477f66e15 P 0fc3233e51734688be72c68053f1415d: Neither blocks nor log segments found. Creating new log.
I20260926 16:48:57.062119 30026 tablet_bootstrap.cc:492] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c: No bootstrap required, opened a new log
I20260926 16:48:57.062391 30026 ts_tablet_manager.cc:1397] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260926 16:48:57.063445 30027 log.cc:824] T ed8519ae4bc6419ca2622f0477f66e15 P 0fc3233e51734688be72c68053f1415d: Log is configured to *not* fsync() on all Append() calls
I20260926 16:48:57.063453 30028 log.cc:824] T ed8519ae4bc6419ca2622f0477f66e15 P 998ecc92cab640f991664783e6c0d19a: Log is configured to *not* fsync() on all Append() calls
I20260926 16:48:57.067133 30027 tablet_bootstrap.cc:492] T ed8519ae4bc6419ca2622f0477f66e15 P 0fc3233e51734688be72c68053f1415d: No bootstrap required, opened a new log
I20260926 16:48:57.067337 30028 tablet_bootstrap.cc:492] T ed8519ae4bc6419ca2622f0477f66e15 P 998ecc92cab640f991664783e6c0d19a: No bootstrap required, opened a new log
I20260926 16:48:57.067528 30027 ts_tablet_manager.cc:1397] T ed8519ae4bc6419ca2622f0477f66e15 P 0fc3233e51734688be72c68053f1415d: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20260926 16:48:57.067706 30028 ts_tablet_manager.cc:1397] T ed8519ae4bc6419ca2622f0477f66e15 P 998ecc92cab640f991664783e6c0d19a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20260926 16:48:57.072686 30026 raft_consensus.cc:348] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998ecc92cab640f991664783e6c0d19a" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41211 } } peers { permanent_uuid: "a7ec24f381ef48ff9a05eaa54af4e21c" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44071 } } peers { permanent_uuid: "0fc3233e51734688be72c68053f1415d" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40795 } }
I20260926 16:48:57.073235 30026 raft_consensus.cc:374] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:48:57.073340 30026 raft_consensus.cc:729] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7ec24f381ef48ff9a05eaa54af4e21c, State: Initialized, Role: FOLLOWER
I20260926 16:48:57.073992 30026 consensus_queue.cc:260] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998ecc92cab640f991664783e6c0d19a" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41211 } } peers { permanent_uuid: "a7ec24f381ef48ff9a05eaa54af4e21c" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44071 } } peers { permanent_uuid: "0fc3233e51734688be72c68053f1415d" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40795 } }
I20260926 16:48:57.075515 29873 heartbeater.cc:503] Master 127.9.238.254:44489 was elected leader, sending a full tablet report...
I20260926 16:48:57.076071 30026 ts_tablet_manager.cc:1428] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
W20260926 16:48:57.077051 29874 tablet.cc:2369] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:48:57.077162 30027 raft_consensus.cc:348] T ed8519ae4bc6419ca2622f0477f66e15 P 0fc3233e51734688be72c68053f1415d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998ecc92cab640f991664783e6c0d19a" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41211 } } peers { permanent_uuid: "a7ec24f381ef48ff9a05eaa54af4e21c" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44071 } } peers { permanent_uuid: "0fc3233e51734688be72c68053f1415d" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40795 } }
I20260926 16:48:57.077692 30027 raft_consensus.cc:374] T ed8519ae4bc6419ca2622f0477f66e15 P 0fc3233e51734688be72c68053f1415d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:48:57.077803 30027 raft_consensus.cc:729] T ed8519ae4bc6419ca2622f0477f66e15 P 0fc3233e51734688be72c68053f1415d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fc3233e51734688be72c68053f1415d, State: Initialized, Role: FOLLOWER
I20260926 16:48:57.078075 30028 raft_consensus.cc:348] T ed8519ae4bc6419ca2622f0477f66e15 P 998ecc92cab640f991664783e6c0d19a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998ecc92cab640f991664783e6c0d19a" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41211 } } peers { permanent_uuid: "a7ec24f381ef48ff9a05eaa54af4e21c" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44071 } } peers { permanent_uuid: "0fc3233e51734688be72c68053f1415d" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40795 } }
I20260926 16:48:57.078503 30028 raft_consensus.cc:374] T ed8519ae4bc6419ca2622f0477f66e15 P 998ecc92cab640f991664783e6c0d19a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:48:57.078572 30028 raft_consensus.cc:729] T ed8519ae4bc6419ca2622f0477f66e15 P 998ecc92cab640f991664783e6c0d19a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 998ecc92cab640f991664783e6c0d19a, State: Initialized, Role: FOLLOWER
I20260926 16:48:57.078660 30027 consensus_queue.cc:260] T ed8519ae4bc6419ca2622f0477f66e15 P 0fc3233e51734688be72c68053f1415d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998ecc92cab640f991664783e6c0d19a" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41211 } } peers { permanent_uuid: "a7ec24f381ef48ff9a05eaa54af4e21c" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44071 } } peers { permanent_uuid: "0fc3233e51734688be72c68053f1415d" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40795 } }
I20260926 16:48:57.079118 30028 consensus_queue.cc:260] T ed8519ae4bc6419ca2622f0477f66e15 P 998ecc92cab640f991664783e6c0d19a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998ecc92cab640f991664783e6c0d19a" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41211 } } peers { permanent_uuid: "a7ec24f381ef48ff9a05eaa54af4e21c" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44071 } } peers { permanent_uuid: "0fc3233e51734688be72c68053f1415d" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40795 } }
I20260926 16:48:57.080194 30007 heartbeater.cc:503] Master 127.9.238.254:44489 was elected leader, sending a full tablet report...
I20260926 16:48:57.081023 30027 ts_tablet_manager.cc:1428] T ed8519ae4bc6419ca2622f0477f66e15 P 0fc3233e51734688be72c68053f1415d: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260926 16:48:57.083276 30028 ts_tablet_manager.cc:1428] T ed8519ae4bc6419ca2622f0477f66e15 P 998ecc92cab640f991664783e6c0d19a: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260926 16:48:57.084149 30032 raft_consensus.cc:482] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:48:57.084571 30032 raft_consensus.cc:504] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998ecc92cab640f991664783e6c0d19a" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41211 } } peers { permanent_uuid: "a7ec24f381ef48ff9a05eaa54af4e21c" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44071 } } peers { permanent_uuid: "0fc3233e51734688be72c68053f1415d" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40795 } }
I20260926 16:48:57.087301 30032 leader_election.cc:288] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 998ecc92cab640f991664783e6c0d19a (127.9.238.193:41211), 0fc3233e51734688be72c68053f1415d (127.9.238.195:40795)
I20260926 16:48:57.096230 30022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:57.087695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:49345 (local address 127.9.238.193:41211)
0926 16:48:57.087972 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:57.087980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:57.088459 (+ 479us) server_negotiation.cc:407] Connection header received
0926 16:48:57.088766 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:57.088772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:57.088893 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:57.088963 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:48:57.089985 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:57.091140 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:57.093649 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:57.095043 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:57.095263 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:57.095354 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:57.095801 (+ 447us) server_negotiation.cc:299] Negotiation successful
0926 16:48:57.095972 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 16:48:57.096782 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:57.088648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:37423 (local address 127.9.238.195:40795)
0926 16:48:57.088824 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:57.088828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:57.090111 (+ 1283us) server_negotiation.cc:407] Connection header received
0926 16:48:57.090376 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:57.090380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:57.090444 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:57.090533 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:48:57.091205 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:57.092130 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:57.094295 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:57.095688 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:57.095916 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:57.095988 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:57.096337 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 16:48:57.096503 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 16:48:57.096751 30036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:57.088399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:40795 (local address 127.9.238.194:37423)
0926 16:48:57.090000 (+ 1601us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:57.090021 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:57.090210 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:57.090588 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:57.090594 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:57.090621 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:48:57.090998 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:57.091009 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:57.092289 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:57.092295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:57.094113 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:57.094124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:57.095945 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:57.095951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:57.096149 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:57.096184 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 16:48:57.096288 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 16:48:57.096420 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1446,"thread_start_us":1188,"threads_started":1}
I20260926 16:48:57.096841 30035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:57.087582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:41211 (local address 127.9.238.194:49345)
0926 16:48:57.088337 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:57.088360 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:57.088590 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:57.089208 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:57.089215 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:57.089245 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:48:57.089785 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:57.089815 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:57.091252 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:57.091258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:57.093419 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:57.093434 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:57.095388 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:57.095393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:57.095537 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:57.095641 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 16:48:57.095730 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 16:48:57.095932 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":1,"thread_start_us":362,"threads_started":1}
I20260926 16:48:57.098316 29694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed8519ae4bc6419ca2622f0477f66e15" candidate_uuid: "a7ec24f381ef48ff9a05eaa54af4e21c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "998ecc92cab640f991664783e6c0d19a" is_pre_election: true
I20260926 16:48:57.098529 29962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed8519ae4bc6419ca2622f0477f66e15" candidate_uuid: "a7ec24f381ef48ff9a05eaa54af4e21c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fc3233e51734688be72c68053f1415d" is_pre_election: true
I20260926 16:48:57.098934 29694 raft_consensus.cc:2454] T ed8519ae4bc6419ca2622f0477f66e15 P 998ecc92cab640f991664783e6c0d19a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7ec24f381ef48ff9a05eaa54af4e21c in term 0.
I20260926 16:48:57.099251 29962 raft_consensus.cc:2454] T ed8519ae4bc6419ca2622f0477f66e15 P 0fc3233e51734688be72c68053f1415d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7ec24f381ef48ff9a05eaa54af4e21c in term 0.
I20260926 16:48:57.099817 29764 leader_election.cc:302] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 998ecc92cab640f991664783e6c0d19a, a7ec24f381ef48ff9a05eaa54af4e21c; no voters:
I20260926 16:48:57.100247 30032 raft_consensus.cc:2781] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:48:57.100474 30032 raft_consensus.cc:482] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:48:57.100632 30032 raft_consensus.cc:3037] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:48:57.103153 30032 raft_consensus.cc:504] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998ecc92cab640f991664783e6c0d19a" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41211 } } peers { permanent_uuid: "a7ec24f381ef48ff9a05eaa54af4e21c" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44071 } } peers { permanent_uuid: "0fc3233e51734688be72c68053f1415d" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40795 } }
I20260926 16:48:57.103976 30032 leader_election.cc:288] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c [CANDIDATE]: Term 1 election: Requested vote from peers 998ecc92cab640f991664783e6c0d19a (127.9.238.193:41211), 0fc3233e51734688be72c68053f1415d (127.9.238.195:40795)
I20260926 16:48:57.104633 29694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed8519ae4bc6419ca2622f0477f66e15" candidate_uuid: "a7ec24f381ef48ff9a05eaa54af4e21c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "998ecc92cab640f991664783e6c0d19a"
I20260926 16:48:57.104883 29694 raft_consensus.cc:3037] T ed8519ae4bc6419ca2622f0477f66e15 P 998ecc92cab640f991664783e6c0d19a [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:48:57.104823 29962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed8519ae4bc6419ca2622f0477f66e15" candidate_uuid: "a7ec24f381ef48ff9a05eaa54af4e21c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fc3233e51734688be72c68053f1415d"
I20260926 16:48:57.105119 29962 raft_consensus.cc:3037] T ed8519ae4bc6419ca2622f0477f66e15 P 0fc3233e51734688be72c68053f1415d [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:48:57.107702 29694 raft_consensus.cc:2454] T ed8519ae4bc6419ca2622f0477f66e15 P 998ecc92cab640f991664783e6c0d19a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7ec24f381ef48ff9a05eaa54af4e21c in term 1.
I20260926 16:48:57.107802 29962 raft_consensus.cc:2454] T ed8519ae4bc6419ca2622f0477f66e15 P 0fc3233e51734688be72c68053f1415d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7ec24f381ef48ff9a05eaa54af4e21c in term 1.
I20260926 16:48:57.108207 29764 leader_election.cc:302] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 998ecc92cab640f991664783e6c0d19a, a7ec24f381ef48ff9a05eaa54af4e21c; no voters:
I20260926 16:48:57.108597 30032 raft_consensus.cc:2781] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:48:57.109220 30032 raft_consensus.cc:686] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c [term 1 LEADER]: Becoming Leader. State: Replica: a7ec24f381ef48ff9a05eaa54af4e21c, State: Running, Role: LEADER
I20260926 16:48:57.109839 30032 consensus_queue.cc:237] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998ecc92cab640f991664783e6c0d19a" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41211 } } peers { permanent_uuid: "a7ec24f381ef48ff9a05eaa54af4e21c" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44071 } } peers { permanent_uuid: "0fc3233e51734688be72c68053f1415d" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40795 } }
I20260926 16:48:57.115944 29549 catalog_manager.cc:5495] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c reported cstate change: term changed from 0 to 1, leader changed from <none> to a7ec24f381ef48ff9a05eaa54af4e21c (127.9.238.194). New cstate: current_term: 1 leader_uuid: "a7ec24f381ef48ff9a05eaa54af4e21c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998ecc92cab640f991664783e6c0d19a" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7ec24f381ef48ff9a05eaa54af4e21c" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0fc3233e51734688be72c68053f1415d" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40795 } health_report { overall_health: UNKNOWN } } }
I20260926 16:48:57.151454 30023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:57.144843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39422 (local address 127.9.238.194:44071)
0926 16:48:57.145157 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:57.145165 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:57.145371 (+ 206us) server_negotiation.cc:407] Connection header received
0926 16:48:57.145594 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:57.145599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:57.145752 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:57.145924 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:48:57.146476 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:57.147309 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:57.148954 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:57.149397 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:57.149504 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:57.150044 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:48:57.150734 (+ 690us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:48:57.150862 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:57.150997 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 16:48:57.151085 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260926 16:48:57.157114 30008 tablet.cc:2369] T ed8519ae4bc6419ca2622f0477f66e15 P 0fc3233e51734688be72c68053f1415d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:48:57.162351 29694 raft_consensus.cc:1261] T ed8519ae4bc6419ca2622f0477f66e15 P 998ecc92cab640f991664783e6c0d19a [term 1 FOLLOWER]: Refusing update from remote peer a7ec24f381ef48ff9a05eaa54af4e21c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:48:57.162595 29962 raft_consensus.cc:1261] T ed8519ae4bc6419ca2622f0477f66e15 P 0fc3233e51734688be72c68053f1415d [term 1 FOLLOWER]: Refusing update from remote peer a7ec24f381ef48ff9a05eaa54af4e21c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:48:57.163733 30037 consensus_queue.cc:1035] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c [LEADER]: Connected to new peer: Peer: permanent_uuid: "998ecc92cab640f991664783e6c0d19a" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 41211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:48:57.164201 30032 consensus_queue.cc:1035] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fc3233e51734688be72c68053f1415d" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 40795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:48:57.179294 30040 mvcc.cc:205] Tried to move back new op lower bound from 7333647717003685888 to 7333647716806754304. Current Snapshot: MvccSnapshot[applied={T|T < 7333647717003685888}]
I20260926 16:48:57.197261 29549 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:48:57.197610 29549 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:48:57.202644 29549 catalog_manager.cc:5782] T 00000000000000000000000000000000 P faf29b2c8f404663823231e6bfc27649: Sending DeleteTablet for 3 replicas of tablet ed8519ae4bc6419ca2622f0477f66e15
I20260926 16:48:57.204579 29808 tablet_service.cc:1514] Processing DeleteTablet for tablet ed8519ae4bc6419ca2622f0477f66e15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:48:57 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:39420
I20260926 16:48:57.204576 29942 tablet_service.cc:1514] Processing DeleteTablet for tablet ed8519ae4bc6419ca2622f0477f66e15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:48:57 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:50844
I20260926 16:48:57.205840 30050 tablet_replica.cc:330] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c: stopping tablet replica
I20260926 16:48:57.205899 29674 tablet_service.cc:1514] Processing DeleteTablet for tablet ed8519ae4bc6419ca2622f0477f66e15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:48:57 UTC) from {username='kudu', principal='kudu/127.9.238.254@KRBTEST.COM'} at 127.0.0.1:59170
I20260926 16:48:57.206828 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 29611
I20260926 16:48:57.206794 30050 raft_consensus.cc:2229] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:48:57.207721 30050 raft_consensus.cc:2258] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:48:57.211871 30050 ts_tablet_manager.cc:1905] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:48:57.220098 30051 tablet_replica.cc:330] T ed8519ae4bc6419ca2622f0477f66e15 P 0fc3233e51734688be72c68053f1415d: stopping tablet replica
I20260926 16:48:57.220731 30051 raft_consensus.cc:2229] T ed8519ae4bc6419ca2622f0477f66e15 P 0fc3233e51734688be72c68053f1415d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:48:57.221309 30051 raft_consensus.cc:2258] T ed8519ae4bc6419ca2622f0477f66e15 P 0fc3233e51734688be72c68053f1415d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:48:57.224112 30051 ts_tablet_manager.cc:1905] T ed8519ae4bc6419ca2622f0477f66e15 P 0fc3233e51734688be72c68053f1415d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:48:57.224567 30052 tablet_replica.cc:330] T ed8519ae4bc6419ca2622f0477f66e15 P 998ecc92cab640f991664783e6c0d19a: stopping tablet replica
I20260926 16:48:57.225230 30052 raft_consensus.cc:2229] T ed8519ae4bc6419ca2622f0477f66e15 P 998ecc92cab640f991664783e6c0d19a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:48:57.226006 30052 raft_consensus.cc:2258] T ed8519ae4bc6419ca2622f0477f66e15 P 998ecc92cab640f991664783e6c0d19a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:48:57.228869 30052 ts_tablet_manager.cc:1905] T ed8519ae4bc6419ca2622f0477f66e15 P 998ecc92cab640f991664783e6c0d19a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:48:57.232136 30051 ts_tablet_manager.cc:1918] T ed8519ae4bc6419ca2622f0477f66e15 P 0fc3233e51734688be72c68053f1415d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:48:57.232340 30051 log.cc:1196] T ed8519ae4bc6419ca2622f0477f66e15 P 0fc3233e51734688be72c68053f1415d: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-2/wal/wals/ed8519ae4bc6419ca2622f0477f66e15
I20260926 16:48:57.232965 30051 ts_tablet_manager.cc:1939] T ed8519ae4bc6419ca2622f0477f66e15 P 0fc3233e51734688be72c68053f1415d: Deleting consensus metadata
I20260926 16:48:57.233330 30050 ts_tablet_manager.cc:1918] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:48:57.233590 30050 log.cc:1196] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-1/wal/wals/ed8519ae4bc6419ca2622f0477f66e15
I20260926 16:48:57.234299 30050 ts_tablet_manager.cc:1939] T ed8519ae4bc6419ca2622f0477f66e15 P a7ec24f381ef48ff9a05eaa54af4e21c: Deleting consensus metadata
I20260926 16:48:57.235272 29537 catalog_manager.cc:4841] TS 0fc3233e51734688be72c68053f1415d (127.9.238.195:40795): tablet ed8519ae4bc6419ca2622f0477f66e15 (table test-table [id=8297bee3b362417083a35cf35e63d0ad]) successfully deleted
I20260926 16:48:57.237367 29534 catalog_manager.cc:4841] TS a7ec24f381ef48ff9a05eaa54af4e21c (127.9.238.194:44071): tablet ed8519ae4bc6419ca2622f0477f66e15 (table test-table [id=8297bee3b362417083a35cf35e63d0ad]) successfully deleted
I20260926 16:48:57.239233 30052 ts_tablet_manager.cc:1918] T ed8519ae4bc6419ca2622f0477f66e15 P 998ecc92cab640f991664783e6c0d19a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:48:57.239504 30052 log.cc:1196] T ed8519ae4bc6419ca2622f0477f66e15 P 998ecc92cab640f991664783e6c0d19a: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790440988577176-10171-0/minicluster-data/ts-0/wal/wals/ed8519ae4bc6419ca2622f0477f66e15
Sep 26 16:48:57 dist-test-slave-wt2j krb5kdc[29505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.193@KRBTEST.COM
I20260926 16:48:57.240275 30052 ts_tablet_manager.cc:1939] T ed8519ae4bc6419ca2622f0477f66e15 P 998ecc92cab640f991664783e6c0d19a: Deleting consensus metadata
I20260926 16:48:57.243149 29537 catalog_manager.cc:4841] TS 998ecc92cab640f991664783e6c0d19a (127.9.238.193:41211): tablet ed8519ae4bc6419ca2622f0477f66e15 (table test-table [id=8297bee3b362417083a35cf35e63d0ad]) successfully deleted
I20260926 16:48:57.250775 30022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:57.208934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59172 (local address 127.9.238.193:41211)
0926 16:48:57.209181 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:57.209191 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:57.220431 (+ 11240us) server_negotiation.cc:407] Connection header received
0926 16:48:57.220909 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:57.220916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:57.221054 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:57.221171 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:57.224178 (+ 3007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:57.225556 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:57.227799 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:57.228531 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:57.228639 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:57.241560 (+ 12921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:57.241587 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:57.241611 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:48:57.241688 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:57.246715 (+ 5027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:57.247930 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:57.247942 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:57.247947 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:57.248152 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:57.248730 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:57.248737 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:57.248742 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:57.249283 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:57.249639 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:57.250044 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 16:48:57.250316 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 16:48:57.251942 29734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:48:57.360399 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 29611
I20260926 16:48:57.388581 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 29745
Sep 26 16:48:57 dist-test-slave-wt2j krb5kdc[29505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.194@KRBTEST.COM
I20260926 16:48:57.410890 30023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:57.389638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39424 (local address 127.9.238.194:44071)
0926 16:48:57.389920 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:57.389928 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:57.389958 (+ 30us) server_negotiation.cc:407] Connection header received
0926 16:48:57.390045 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:57.390052 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:57.390146 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:57.390313 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:57.391862 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:57.393277 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:57.394760 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:57.395732 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:57.395885 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:57.401761 (+ 5876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:57.401785 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:57.401809 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:48:57.401873 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:57.406651 (+ 4778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:57.407977 (+ 1326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:57.407988 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:57.407992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:57.408208 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:57.408774 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:57.408780 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:57.408784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:57.409362 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:57.409719 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:57.410119 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 16:48:57.410389 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260926 16:48:57.411911 29868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:48:57.532232 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 29745
I20260926 16:48:57.549507 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 29879
Sep 26 16:48:57 dist-test-slave-wt2j krb5kdc[29505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.195@KRBTEST.COM
I20260926 16:48:57.568367 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:57.550586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50852 (local address 127.9.238.195:40795)
0926 16:48:57.550810 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:57.550815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:57.550837 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:48:57.551050 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:57.551056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:57.551142 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:57.551244 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:57.552902 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:57.553688 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:57.555150 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:57.555625 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:57.555751 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:57.561001 (+ 5250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:57.561033 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:57.561056 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:48:57.561112 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:57.564829 (+ 3717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:57.565839 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:57.565849 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:57.565852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:57.566030 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:48:57.566497 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:48:57.566501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:48:57.566503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:48:57.567010 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:57.567301 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:57.567793 (+ 492us) server_negotiation.cc:299] Negotiation successful
0926 16:48:57.568042 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 16:48:57.569257 30002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:48:57.679844 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 29879
I20260926 16:48:57.700173 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 29518
I20260926 16:48:57.701745 29580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:48:57.820035 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 29518
2026-09-26T16:48:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4810 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T16:48:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:48:57Z Disabled control of system clock
I20260926 16:48:57.920020 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:44677
--webserver_interface=127.9.238.254
--webserver_port=0
--builtin_ntp_servers=127.9.238.212:40949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:44677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 16:48:58.264401 30061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:48:58.264693 30061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:48:58.264824 30061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:48:58.272164 30061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:48:58.272277 30061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:48:58.272341 30061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:48:58.272441 30061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:48:58.272513 30061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:48:58.281813 30061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:40949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:44677
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:44677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:48:58.283183 30061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:48:58.285089 30061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:48:58.298209 30066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:58.298327 30067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:58.298774 30069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:48:58.300498 30061 server_base.cc:1034] running on GCE node
I20260926 16:48:58.301653 30061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:48:58.303853 30061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:48:58.305205 30061 hybrid_clock.cc:648] HybridClock initialized: now 1790441338305135 us; error 45 us; skew 500 ppm
I20260926 16:48:58.305747 30061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:48:58.309141 30061 webserver.cc:460] Webserver started at http://127.9.238.254:40451/ using document root <none> and password file <none>
I20260926 16:48:58.310606 30061 fs_manager.cc:359] Metadata directory not provided
I20260926 16:48:58.310812 30061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:48:58.311295 30061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:48:58.315716 30061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "95a5467ad64349a280ae0fcba7ab02dc"
format_stamp: "Formatted at 2026-09-26 16:48:58 on dist-test-slave-wt2j"
server_key: "0dfdc0114ac324a6b829bbaa8c3f7b60"
server_key_iv: "88ad16b9693cf3abe66826194fd59e56"
server_key_version: "encryptionkey@0"
I20260926 16:48:58.316872 30061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "95a5467ad64349a280ae0fcba7ab02dc"
format_stamp: "Formatted at 2026-09-26 16:48:58 on dist-test-slave-wt2j"
server_key: "0dfdc0114ac324a6b829bbaa8c3f7b60"
server_key_iv: "88ad16b9693cf3abe66826194fd59e56"
server_key_version: "encryptionkey@0"
I20260926 16:48:58.324142 30061 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260926 16:48:58.329730 30075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:58.331736 30061 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.007s sys 0.000s
I20260926 16:48:58.332069 30061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "95a5467ad64349a280ae0fcba7ab02dc"
format_stamp: "Formatted at 2026-09-26 16:48:58 on dist-test-slave-wt2j"
server_key: "0dfdc0114ac324a6b829bbaa8c3f7b60"
server_key_iv: "88ad16b9693cf3abe66826194fd59e56"
server_key_version: "encryptionkey@0"
I20260926 16:48:58.332397 30061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:48:58.358620 30061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:48:58.359851 30061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:48:58.360224 30061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:48:58.384066 30061 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:44677
I20260926 16:48:58.384117 30126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:44677 every 8 connection(s)
I20260926 16:48:58.385828 30061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:48:58.387403 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 30061
I20260926 16:48:58.387650 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:48:58.388216 10171 external_mini_cluster.cc:1428] Setting key 27d7ea3b60e90e8c92039180a615514a
I20260926 16:48:58.391278 30127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:48:58.406281 30127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95a5467ad64349a280ae0fcba7ab02dc: Bootstrap starting.
I20260926 16:48:58.406306 30130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:58.390607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59528 (local address 127.9.238.254:44677)
0926 16:48:58.391985 (+ 1378us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:58.392000 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:58.392057 (+ 57us) server_negotiation.cc:407] Connection header received
0926 16:48:58.394044 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:58.394074 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:58.394709 (+ 635us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:58.395158 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:58.396121 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:58.397667 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:58.399766 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:58.400837 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:58.401007 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:58.402707 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:58.402718 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:58.402750 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:48:58.403833 (+ 1083us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:58.403976 (+ 143us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:48:58.404026 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:58.404162 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:58.404467 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 16:48:58.404982 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":445,"threads_started":1}
I20260926 16:48:58.410704 30127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95a5467ad64349a280ae0fcba7ab02dc: Neither blocks nor log segments found. Creating new log.
I20260926 16:48:58.411934 30127 log.cc:824] T 00000000000000000000000000000000 P 95a5467ad64349a280ae0fcba7ab02dc: Log is configured to *not* fsync() on all Append() calls
I20260926 16:48:58.415897 30127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95a5467ad64349a280ae0fcba7ab02dc: No bootstrap required, opened a new log
I20260926 16:48:58.423789 30127 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95a5467ad64349a280ae0fcba7ab02dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a5467ad64349a280ae0fcba7ab02dc" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44677 } }
I20260926 16:48:58.424175 30127 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95a5467ad64349a280ae0fcba7ab02dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:48:58.424304 30127 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95a5467ad64349a280ae0fcba7ab02dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a5467ad64349a280ae0fcba7ab02dc, State: Initialized, Role: FOLLOWER
I20260926 16:48:58.424947 30127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95a5467ad64349a280ae0fcba7ab02dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a5467ad64349a280ae0fcba7ab02dc" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44677 } }
I20260926 16:48:58.425185 30127 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95a5467ad64349a280ae0fcba7ab02dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:48:58.425349 30127 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95a5467ad64349a280ae0fcba7ab02dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:48:58.425540 30127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95a5467ad64349a280ae0fcba7ab02dc [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:48:58.427819 30127 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95a5467ad64349a280ae0fcba7ab02dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a5467ad64349a280ae0fcba7ab02dc" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44677 } }
I20260926 16:48:58.428361 30127 leader_election.cc:302] T 00000000000000000000000000000000 P 95a5467ad64349a280ae0fcba7ab02dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95a5467ad64349a280ae0fcba7ab02dc; no voters:
I20260926 16:48:58.429136 30127 leader_election.cc:288] T 00000000000000000000000000000000 P 95a5467ad64349a280ae0fcba7ab02dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:48:58.429590 30132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95a5467ad64349a280ae0fcba7ab02dc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:48:58.430791 30132 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95a5467ad64349a280ae0fcba7ab02dc [term 1 LEADER]: Becoming Leader. State: Replica: 95a5467ad64349a280ae0fcba7ab02dc, State: Running, Role: LEADER
I20260926 16:48:58.431751 30127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95a5467ad64349a280ae0fcba7ab02dc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:48:58.431586 30132 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95a5467ad64349a280ae0fcba7ab02dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a5467ad64349a280ae0fcba7ab02dc" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44677 } }
I20260926 16:48:58.436219 30134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95a5467ad64349a280ae0fcba7ab02dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95a5467ad64349a280ae0fcba7ab02dc. Latest consensus state: current_term: 1 leader_uuid: "95a5467ad64349a280ae0fcba7ab02dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a5467ad64349a280ae0fcba7ab02dc" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44677 } } }
I20260926 16:48:58.437069 30134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95a5467ad64349a280ae0fcba7ab02dc [sys.catalog]: This master's current role is: LEADER
I20260926 16:48:58.436223 30133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95a5467ad64349a280ae0fcba7ab02dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95a5467ad64349a280ae0fcba7ab02dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a5467ad64349a280ae0fcba7ab02dc" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44677 } } }
I20260926 16:48:58.437618 30133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95a5467ad64349a280ae0fcba7ab02dc [sys.catalog]: This master's current role is: LEADER
I20260926 16:48:58.445405 30142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:48:58.450246 30142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:48:58.461031 30142 catalog_manager.cc:1347] Generated new cluster ID: 17dff01b6bd64e05be291c59f37e1d55
I20260926 16:48:58.461205 30142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:48:58.493114 30142 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:48:58.494700 30142 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:48:58.504447 30142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95a5467ad64349a280ae0fcba7ab02dc: Generated new TSK 0
I20260926 16:48:58.505641 30142 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:48:58.518311 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.193:0
--local_ip_for_outbound_sockets=127.9.238.193
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44677
--builtin_ntp_servers=127.9.238.212:40949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 16:48:58.906891 30151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:48:58.907246 30151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:48:58.907390 30151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:48:58.915133 30151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:48:58.915282 30151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:48:58.915448 30151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.193
I20260926 16:48:58.925119 30151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:40949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.9.238.193
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.193
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:48:58.926469 30151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:48:58.928272 30151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:48:58.943109 30156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:58.943081 30157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:58.943763 30159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:48:58.945309 30151 server_base.cc:1034] running on GCE node
I20260926 16:48:58.945995 30151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:48:58.947026 30151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:48:58.948251 30151 hybrid_clock.cc:648] HybridClock initialized: now 1790441338948188 us; error 58 us; skew 500 ppm
I20260926 16:48:58.948709 30151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:48:58.951538 30151 webserver.cc:460] Webserver started at http://127.9.238.193:33671/ using document root <none> and password file <none>
I20260926 16:48:58.952332 30151 fs_manager.cc:359] Metadata directory not provided
I20260926 16:48:58.952428 30151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:48:58.952699 30151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:48:58.955695 30151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-0/data/instance:
uuid: "78ead6c10c80474487ada86645614da1"
format_stamp: "Formatted at 2026-09-26 16:48:58 on dist-test-slave-wt2j"
server_key: "5609558f6125cfd5114258901ff04e41"
server_key_iv: "fb5372ab9a8e3a010fe84b209d8a8e0e"
server_key_version: "encryptionkey@0"
I20260926 16:48:58.956431 30151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance:
uuid: "78ead6c10c80474487ada86645614da1"
format_stamp: "Formatted at 2026-09-26 16:48:58 on dist-test-slave-wt2j"
server_key: "5609558f6125cfd5114258901ff04e41"
server_key_iv: "fb5372ab9a8e3a010fe84b209d8a8e0e"
server_key_version: "encryptionkey@0"
I20260926 16:48:58.962040 30151 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:48:58.966128 30165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:58.967994 30151 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 16:48:58.968194 30151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-0/wal
uuid: "78ead6c10c80474487ada86645614da1"
format_stamp: "Formatted at 2026-09-26 16:48:58 on dist-test-slave-wt2j"
server_key: "5609558f6125cfd5114258901ff04e41"
server_key_iv: "fb5372ab9a8e3a010fe84b209d8a8e0e"
server_key_version: "encryptionkey@0"
I20260926 16:48:58.968413 30151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:48:59.005110 30151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:48:59.006007 30151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:48:59.006343 30151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:48:59.007490 30151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:48:59.009105 30151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:48:59.009204 30151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:59.009349 30151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:48:59.009403 30151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:59.050527 30151 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.193:44561
I20260926 16:48:59.050562 30277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.193:44561 every 8 connection(s)
I20260926 16:48:59.051965 30151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-0/data/info.pb
I20260926 16:48:59.056783 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 30151
I20260926 16:48:59.057001 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-0/wal/instance
I20260926 16:48:59.057667 10171 external_mini_cluster.cc:1428] Setting key 7c237fa54b0fe5ff3b6872ba35da646b
I20260926 16:48:59.061417 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.194:0
--local_ip_for_outbound_sockets=127.9.238.194
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44677
--builtin_ntp_servers=127.9.238.212:40949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 16:48:59.070132 30280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:59.055398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:47925 (local address 127.9.238.254:44677)
0926 16:48:59.056069 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:59.056075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:59.057762 (+ 1687us) server_negotiation.cc:407] Connection header received
0926 16:48:59.060569 (+ 2807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:59.060573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:59.060644 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:59.060757 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:59.062675 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:59.063454 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:59.066423 (+ 2969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:59.066863 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:59.066978 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:59.069140 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:59.069145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:59.069148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:48:59.069170 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:59.069220 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:48:59.069233 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:59.069321 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:59.069701 (+ 380us) server_negotiation.cc:299] Negotiation successful
0926 16:48:59.069793 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":387,"threads_started":1}
I20260926 16:48:59.071931 30281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:59.055960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:44677 (local address 127.9.238.193:47925)
0926 16:48:59.057466 (+ 1506us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:59.057532 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:59.060169 (+ 2637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:59.061125 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:59.061142 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:59.061324 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:48:59.062419 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:59.062451 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:59.063685 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:59.063696 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:59.066216 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:59.066239 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:59.067157 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:59.067166 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:59.067601 (+ 435us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:59.068634 (+ 1033us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:48:59.068640 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:48:59.068736 (+ 96us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:48:59.068893 (+ 157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:48:59.069355 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:48:59.069369 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:48:59.069382 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 16:48:59.069615 (+ 233us) client_negotiation.cc:240] Negotiation successful
0926 16:48:59.070186 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":388,"threads_started":1}
I20260926 16:48:59.074194 30278 heartbeater.cc:347] Connected to a master server at 127.9.238.254:44677
I20260926 16:48:59.074654 30278 heartbeater.cc:464] Registering TS with master...
I20260926 16:48:59.075781 30278 heartbeater.cc:511] Master 127.9.238.254:44677 requested a full tablet report, sending...
I20260926 16:48:59.078864 30092 ts_manager.cc:194] Registered new tserver with Master: 78ead6c10c80474487ada86645614da1 (127.9.238.193:44561)
I20260926 16:48:59.080741 30092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.238.193:47925
W20260926 16:48:59.397598 30282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:48:59.397881 30282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:48:59.398013 30282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:48:59.405474 30282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:48:59.405589 30282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:48:59.405752 30282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.194
I20260926 16:48:59.415277 30282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:40949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.9.238.194
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.194
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:48:59.416631 30282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:48:59.418344 30282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:48:59.432852 30287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:59.432828 30288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:48:59.432750 30290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:48:59.433648 30282 server_base.cc:1034] running on GCE node
I20260926 16:48:59.434329 30282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:48:59.435448 30282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:48:59.436721 30282 hybrid_clock.cc:648] HybridClock initialized: now 1790441339436667 us; error 57 us; skew 500 ppm
I20260926 16:48:59.437090 30282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:48:59.439657 30282 webserver.cc:460] Webserver started at http://127.9.238.194:44241/ using document root <none> and password file <none>
I20260926 16:48:59.440392 30282 fs_manager.cc:359] Metadata directory not provided
I20260926 16:48:59.440523 30282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:48:59.440833 30282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:48:59.443797 30282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-1/data/instance:
uuid: "5981cbda4370475db447a8ece9df929f"
format_stamp: "Formatted at 2026-09-26 16:48:59 on dist-test-slave-wt2j"
server_key: "54f2d86f7bdb262ba452ab75935ba12a"
server_key_iv: "ff7a852d755dc7a1c0b7dd9c48b5cccd"
server_key_version: "encryptionkey@0"
I20260926 16:48:59.444612 30282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance:
uuid: "5981cbda4370475db447a8ece9df929f"
format_stamp: "Formatted at 2026-09-26 16:48:59 on dist-test-slave-wt2j"
server_key: "54f2d86f7bdb262ba452ab75935ba12a"
server_key_iv: "ff7a852d755dc7a1c0b7dd9c48b5cccd"
server_key_version: "encryptionkey@0"
I20260926 16:48:59.450178 30282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260926 16:48:59.454145 30296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:59.456095 30282 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:48:59.456274 30282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-1/wal
uuid: "5981cbda4370475db447a8ece9df929f"
format_stamp: "Formatted at 2026-09-26 16:48:59 on dist-test-slave-wt2j"
server_key: "54f2d86f7bdb262ba452ab75935ba12a"
server_key_iv: "ff7a852d755dc7a1c0b7dd9c48b5cccd"
server_key_version: "encryptionkey@0"
I20260926 16:48:59.456494 30282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:48:59.480042 30282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:48:59.480888 30282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:48:59.481180 30282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:48:59.482314 30282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:48:59.484153 30282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:48:59.484251 30282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:59.484328 30282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:48:59.484402 30282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:48:59.525585 30282 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.194:44397
I20260926 16:48:59.525753 30408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.194:44397 every 8 connection(s)
I20260926 16:48:59.526976 30282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-1/data/info.pb
I20260926 16:48:59.536461 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 30282
I20260926 16:48:59.536684 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-1/wal/instance
I20260926 16:48:59.537289 10171 external_mini_cluster.cc:1428] Setting key 7ed8f24551f10c018e78815fb9718b00
I20260926 16:48:59.541326 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.9.238.195:0
--local_ip_for_outbound_sockets=127.9.238.195
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44677
--builtin_ntp_servers=127.9.238.212:40949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 16:48:59.542035 30280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:59.530346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.194:47577 (local address 127.9.238.254:44677)
0926 16:48:59.530718 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0926 16:48:59.530724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:48:59.532525 (+ 1801us) server_negotiation.cc:407] Connection header received
0926 16:48:59.534406 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:48:59.534413 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:48:59.534539 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:48:59.534673 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:48:59.535858 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:59.536672 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:59.538698 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:48:59.539186 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:48:59.539325 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:59.541099 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:48:59.541105 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:48:59.541108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:48:59.541130 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:48:59.541165 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:48:59.541179 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:48:59.541290 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 16:48:59.541554 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 16:48:59.541665 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260926 16:48:59.543179 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:48:59.530920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:44677 (local address 127.9.238.194:47577)
0926 16:48:59.532299 (+ 1379us) negotiation.cc:107] Waiting for socket to connect
0926 16:48:59.532343 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 16:48:59.534119 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:48:59.534917 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:48:59.534925 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:48:59.535026 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:48:59.535698 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:59.535718 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:59.536860 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:59.536864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:59.538538 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:48:59.538551 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:48:59.539429 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:48:59.539437 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:48:59.539747 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:48:59.540711 (+ 964us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:48:59.540714 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:48:59.540765 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:48:59.540854 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:48:59.541296 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:48:59.541310 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:48:59.541322 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 16:48:59.541520 (+ 198us) client_negotiation.cc:240] Negotiation successful
0926 16:48:59.541981 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":526,"threads_started":1}
I20260926 16:48:59.549453 30409 heartbeater.cc:347] Connected to a master server at 127.9.238.254:44677
I20260926 16:48:59.549993 30409 heartbeater.cc:464] Registering TS with master...
I20260926 16:48:59.551307 30409 heartbeater.cc:511] Master 127.9.238.254:44677 requested a full tablet report, sending...
I20260926 16:48:59.553418 30092 ts_manager.cc:194] Registered new tserver with Master: 5981cbda4370475db447a8ece9df929f (127.9.238.194:44397)
I20260926 16:48:59.554390 30092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.238.194:47577
W20260926 16:48:59.964234 30412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:48:59.964659 30412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:48:59.964835 30412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:48:59.973342 30412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:48:59.973516 30412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:48:59.973699 30412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.9.238.195
I20260926 16:48:59.984632 30412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:40949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.9.238.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.9.238.195
--webserver_port=0
--tserver_master_addrs=127.9.238.254:44677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.9.238.195
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:48:59.986464 30412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:48:59.988854 30412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:49:00.006662 30417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:00.006760 30418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:00.007578 30420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:49:00.009912 30412 server_base.cc:1034] running on GCE node
I20260926 16:49:00.011104 30412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:49:00.012650 30412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:49:00.013949 30412 hybrid_clock.cc:648] HybridClock initialized: now 1790441340013894 us; error 39 us; skew 500 ppm
I20260926 16:49:00.014436 30412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260926 16:49:00.024237 30412 webserver.cc:460] Webserver started at http://127.9.238.195:39295/ using document root <none> and password file <none>
I20260926 16:49:00.025264 30412 fs_manager.cc:359] Metadata directory not provided
I20260926 16:49:00.025379 30412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:49:00.025748 30412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:49:00.029367 30412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-2/data/instance:
uuid: "6e6f27df3479439ea7ba80103602b144"
format_stamp: "Formatted at 2026-09-26 16:49:00 on dist-test-slave-wt2j"
server_key: "462a6664637704bc2ef3404647734ca6"
server_key_iv: "2d57696e777df038bed7ff5c6716e359"
server_key_version: "encryptionkey@0"
I20260926 16:49:00.030195 30412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance:
uuid: "6e6f27df3479439ea7ba80103602b144"
format_stamp: "Formatted at 2026-09-26 16:49:00 on dist-test-slave-wt2j"
server_key: "462a6664637704bc2ef3404647734ca6"
server_key_iv: "2d57696e777df038bed7ff5c6716e359"
server_key_version: "encryptionkey@0"
I20260926 16:49:00.036667 30412 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 16:49:00.041105 30426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:49:00.043012 30412 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 16:49:00.043202 30412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-2/wal
uuid: "6e6f27df3479439ea7ba80103602b144"
format_stamp: "Formatted at 2026-09-26 16:49:00 on dist-test-slave-wt2j"
server_key: "462a6664637704bc2ef3404647734ca6"
server_key_iv: "2d57696e777df038bed7ff5c6716e359"
server_key_version: "encryptionkey@0"
I20260926 16:49:00.043421 30412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:49:00.064536 30412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:49:00.065407 30412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:49:00.065677 30412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:49:00.066721 30412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 16:49:00.068378 30412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 16:49:00.068472 30412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 16:49:00.068547 30412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 16:49:00.068608 30412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 16:49:00.085529 30278 heartbeater.cc:503] Master 127.9.238.254:44677 was elected leader, sending a full tablet report...
I20260926 16:49:00.109743 30412 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.195:44497
I20260926 16:49:00.109858 30538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.195:44497 every 8 connection(s)
I20260926 16:49:00.111217 30412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-2/data/info.pb
I20260926 16:49:00.116986 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 30412
I20260926 16:49:00.117208 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-2/wal/instance
I20260926 16:49:00.117764 10171 external_mini_cluster.cc:1428] Setting key 6c004c4e495d2e9604d96a6c6d59668c
I20260926 16:49:00.126240 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:00.114656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.195:46045 (local address 127.9.238.254:44677)
0926 16:49:00.115256 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:00.115264 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:00.117106 (+ 1842us) server_negotiation.cc:407] Connection header received
0926 16:49:00.119413 (+ 2307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:00.119419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:00.119522 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:00.119692 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:00.120846 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.121605 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.123245 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.123747 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.123883 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:00.125324 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:00.125329 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:00.125338 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:00.125359 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:00.125402 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:00.125414 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:00.125513 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:00.125748 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 16:49:00.125860 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":263,"threads_started":1}
I20260926 16:49:00.127374 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:00.115409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:44677 (local address 127.9.238.195:46045)
0926 16:49:00.116787 (+ 1378us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:00.116859 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 16:49:00.119071 (+ 2212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:49:00.119945 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:49:00.119954 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:49:00.120062 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:49:00.120669 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:00.120686 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.121762 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.121772 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:00.123091 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:00.123104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.123986 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.123992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:00.124248 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:00.124986 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 16:49:00.124989 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:49:00.125035 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 16:49:00.125126 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:49:00.125522 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:49:00.125534 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:49:00.125544 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 16:49:00.125737 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 16:49:00.126169 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":414,"threads_started":1}
I20260926 16:49:00.129379 30539 heartbeater.cc:347] Connected to a master server at 127.9.238.254:44677
I20260926 16:49:00.129729 30539 heartbeater.cc:464] Registering TS with master...
I20260926 16:49:00.130556 30539 heartbeater.cc:511] Master 127.9.238.254:44677 requested a full tablet report, sending...
I20260926 16:49:00.132431 30092 ts_manager.cc:194] Registered new tserver with Master: 6e6f27df3479439ea7ba80103602b144 (127.9.238.195:44497)
I20260926 16:49:00.133368 30092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.9.238.195:46045
I20260926 16:49:00.134433 10171 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 16:49:00.149430 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:00.141214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59530 (local address 127.9.238.254:44677)
0926 16:49:00.141570 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:00.141577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:00.141888 (+ 311us) server_negotiation.cc:407] Connection header received
0926 16:49:00.142182 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:00.142187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:00.142260 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:00.142379 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:00.142973 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.144201 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.146064 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.146696 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.146885 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:00.148536 (+ 1651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:00.148542 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:00.148545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:00.148572 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:00.148617 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:00.148633 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:00.148742 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:00.148927 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 16:49:00.149060 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260926 16:49:00.155817 30091 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 16:49:00.164438 30091 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 16:49:00.219759 30556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:00.180099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:44497 (local address 127.0.0.1:40460)
0926 16:49:00.207122 (+ 27023us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:00.207195 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 16:49:00.207488 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:49:00.209935 (+ 2447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:49:00.209947 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:49:00.209996 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:49:00.210720 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:00.210741 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.213557 (+ 2816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.213562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:00.216101 (+ 2539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:00.216119 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.218645 (+ 2526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.218652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:00.218879 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:00.218951 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 16:49:00.219109 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 16:49:00.219328 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26575,"thread_start_us":29843,"threads_started":1}
I20260926 16:49:00.219956 30555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:00.180815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40460 (local address 127.9.238.195:44497)
0926 16:49:00.206124 (+ 25309us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:00.206139 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:00.208712 (+ 2573us) server_negotiation.cc:407] Connection header received
0926 16:49:00.208853 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:00.208865 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:00.209220 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:00.209474 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:49:00.211149 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.213232 (+ 2083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.216411 (+ 3179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.218378 (+ 1967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.218565 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:00.218987 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:00.219167 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 16:49:00.219413 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25015,"thread_start_us":27560,"threads_started":1}
I20260926 16:49:00.221202 30552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:00.180113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:44397 (local address 127.0.0.1:51274)
0926 16:49:00.210440 (+ 30327us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:00.210473 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 16:49:00.210696 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:49:00.212400 (+ 1704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:49:00.212406 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:49:00.212441 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:49:00.213621 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:00.213638 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.215658 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.215664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:00.218264 (+ 2600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:00.218278 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.220558 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.220565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:00.220762 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:00.220800 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 16:49:00.220886 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 16:49:00.220990 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30106,"thread_start_us":30519,"threads_started":1}
I20260926 16:49:00.221990 30554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:00.180539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51274 (local address 127.9.238.194:44397)
0926 16:49:00.211602 (+ 31063us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:00.211610 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:00.211641 (+ 31us) server_negotiation.cc:407] Connection header received
0926 16:49:00.211717 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:00.211722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:00.211961 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:00.212154 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:49:00.213886 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.215428 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.218456 (+ 3028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.220011 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.220154 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:00.220416 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:00.221454 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0926 16:49:00.221624 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30854,"thread_start_us":30725,"threads_started":1}
I20260926 16:49:00.224232 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:00.182143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57478 (local address 127.9.238.193:44561)
0926 16:49:00.208343 (+ 26200us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:00.208356 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:00.211174 (+ 2818us) server_negotiation.cc:407] Connection header received
0926 16:49:00.211343 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:00.211356 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:00.211735 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:00.212014 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:49:00.213737 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.215946 (+ 2209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.219887 (+ 3941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.221683 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.221863 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:00.222190 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:00.223740 (+ 1550us) server_negotiation.cc:299] Negotiation successful
0926 16:49:00.223918 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25941,"thread_start_us":27231,"threads_started":1}
I20260926 16:49:00.224758 30553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:00.181111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:44561 (local address 127.0.0.1:57478)
0926 16:49:00.210863 (+ 29752us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:00.210887 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 16:49:00.211031 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:49:00.212310 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:49:00.212316 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:49:00.212346 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:49:00.213515 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:00.213534 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.216148 (+ 2614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.216156 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:00.219474 (+ 3318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:00.219496 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.222375 (+ 2879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.222384 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:00.222589 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:00.222630 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 16:49:00.222761 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 16:49:00.222939 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29537,"mutex_wait_us":1,"spinlock_wait_cycles":3456,"thread_start_us":25313,"threads_started":1}
I20260926 16:49:00.226123 30474 tablet_service.cc:1467] Processing CreateTablet for tablet 9347a6b0031d4a5ba34e2fea3ade2ce0 (DEFAULT_TABLE table=test-table [id=139606a5fbe24586b7305b816b9d2287]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:49:00.226125 30344 tablet_service.cc:1467] Processing CreateTablet for tablet 9347a6b0031d4a5ba34e2fea3ade2ce0 (DEFAULT_TABLE table=test-table [id=139606a5fbe24586b7305b816b9d2287]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:49:00.228178 30344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9347a6b0031d4a5ba34e2fea3ade2ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:49:00.228178 30474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9347a6b0031d4a5ba34e2fea3ade2ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:49:00.228273 30213 tablet_service.cc:1467] Processing CreateTablet for tablet 9347a6b0031d4a5ba34e2fea3ade2ce0 (DEFAULT_TABLE table=test-table [id=139606a5fbe24586b7305b816b9d2287]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 16:49:00.229813 30213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9347a6b0031d4a5ba34e2fea3ade2ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:49:00.241964 30558 tablet_bootstrap.cc:492] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1: Bootstrap starting.
I20260926 16:49:00.245647 30558 tablet_bootstrap.cc:654] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1: Neither blocks nor log segments found. Creating new log.
I20260926 16:49:00.246757 30558 log.cc:824] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1: Log is configured to *not* fsync() on all Append() calls
I20260926 16:49:00.252790 30559 tablet_bootstrap.cc:492] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 5981cbda4370475db447a8ece9df929f: Bootstrap starting.
I20260926 16:49:00.253360 30560 tablet_bootstrap.cc:492] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 6e6f27df3479439ea7ba80103602b144: Bootstrap starting.
I20260926 16:49:00.258219 30559 tablet_bootstrap.cc:654] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 5981cbda4370475db447a8ece9df929f: Neither blocks nor log segments found. Creating new log.
I20260926 16:49:00.259626 30559 log.cc:824] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 5981cbda4370475db447a8ece9df929f: Log is configured to *not* fsync() on all Append() calls
I20260926 16:49:00.260660 30560 tablet_bootstrap.cc:654] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 6e6f27df3479439ea7ba80103602b144: Neither blocks nor log segments found. Creating new log.
I20260926 16:49:00.260742 30558 tablet_bootstrap.cc:492] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1: No bootstrap required, opened a new log
I20260926 16:49:00.261075 30558 ts_tablet_manager.cc:1397] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.008s
I20260926 16:49:00.262080 30560 log.cc:824] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 6e6f27df3479439ea7ba80103602b144: Log is configured to *not* fsync() on all Append() calls
I20260926 16:49:00.263291 30559 tablet_bootstrap.cc:492] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 5981cbda4370475db447a8ece9df929f: No bootstrap required, opened a new log
I20260926 16:49:00.263602 30559 ts_tablet_manager.cc:1397] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 5981cbda4370475db447a8ece9df929f: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260926 16:49:00.265249 30560 tablet_bootstrap.cc:492] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 6e6f27df3479439ea7ba80103602b144: No bootstrap required, opened a new log
I20260926 16:49:00.265588 30560 ts_tablet_manager.cc:1397] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 6e6f27df3479439ea7ba80103602b144: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260926 16:49:00.272083 30558 raft_consensus.cc:348] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6f27df3479439ea7ba80103602b144" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44497 } } peers { permanent_uuid: "5981cbda4370475db447a8ece9df929f" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44397 } } peers { permanent_uuid: "78ead6c10c80474487ada86645614da1" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44561 } }
I20260926 16:49:00.272593 30558 raft_consensus.cc:374] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:49:00.272720 30558 raft_consensus.cc:729] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78ead6c10c80474487ada86645614da1, State: Initialized, Role: FOLLOWER
I20260926 16:49:00.273439 30558 consensus_queue.cc:260] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6f27df3479439ea7ba80103602b144" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44497 } } peers { permanent_uuid: "5981cbda4370475db447a8ece9df929f" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44397 } } peers { permanent_uuid: "78ead6c10c80474487ada86645614da1" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44561 } }
I20260926 16:49:00.274238 30559 raft_consensus.cc:348] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 5981cbda4370475db447a8ece9df929f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6f27df3479439ea7ba80103602b144" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44497 } } peers { permanent_uuid: "5981cbda4370475db447a8ece9df929f" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44397 } } peers { permanent_uuid: "78ead6c10c80474487ada86645614da1" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44561 } }
I20260926 16:49:00.274725 30559 raft_consensus.cc:374] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 5981cbda4370475db447a8ece9df929f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:49:00.274812 30559 raft_consensus.cc:729] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 5981cbda4370475db447a8ece9df929f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5981cbda4370475db447a8ece9df929f, State: Initialized, Role: FOLLOWER
I20260926 16:49:00.275491 30558 ts_tablet_manager.cc:1428] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260926 16:49:00.275488 30559 consensus_queue.cc:260] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 5981cbda4370475db447a8ece9df929f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6f27df3479439ea7ba80103602b144" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44497 } } peers { permanent_uuid: "5981cbda4370475db447a8ece9df929f" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44397 } } peers { permanent_uuid: "78ead6c10c80474487ada86645614da1" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44561 } }
I20260926 16:49:00.276350 30560 raft_consensus.cc:348] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 6e6f27df3479439ea7ba80103602b144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6f27df3479439ea7ba80103602b144" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44497 } } peers { permanent_uuid: "5981cbda4370475db447a8ece9df929f" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44397 } } peers { permanent_uuid: "78ead6c10c80474487ada86645614da1" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44561 } }
I20260926 16:49:00.276806 30409 heartbeater.cc:503] Master 127.9.238.254:44677 was elected leader, sending a full tablet report...
I20260926 16:49:00.276904 30560 raft_consensus.cc:374] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 6e6f27df3479439ea7ba80103602b144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:49:00.277011 30560 raft_consensus.cc:729] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 6e6f27df3479439ea7ba80103602b144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e6f27df3479439ea7ba80103602b144, State: Initialized, Role: FOLLOWER
I20260926 16:49:00.277518 30559 ts_tablet_manager.cc:1428] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 5981cbda4370475db447a8ece9df929f: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
I20260926 16:49:00.277747 30560 consensus_queue.cc:260] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 6e6f27df3479439ea7ba80103602b144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6f27df3479439ea7ba80103602b144" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44497 } } peers { permanent_uuid: "5981cbda4370475db447a8ece9df929f" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44397 } } peers { permanent_uuid: "78ead6c10c80474487ada86645614da1" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44561 } }
W20260926 16:49:00.279157 30410 tablet.cc:2369] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 5981cbda4370475db447a8ece9df929f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:49:00.279379 30539 heartbeater.cc:503] Master 127.9.238.254:44677 was elected leader, sending a full tablet report...
I20260926 16:49:00.279917 30560 ts_tablet_manager.cc:1428] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 6e6f27df3479439ea7ba80103602b144: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
W20260926 16:49:00.304946 30279 tablet.cc:2369] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:49:00.363317 30540 tablet.cc:2369] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 6e6f27df3479439ea7ba80103602b144: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:49:00.594807 30564 raft_consensus.cc:482] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 16:49:00.595170 30564 raft_consensus.cc:504] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6f27df3479439ea7ba80103602b144" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44497 } } peers { permanent_uuid: "5981cbda4370475db447a8ece9df929f" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44397 } } peers { permanent_uuid: "78ead6c10c80474487ada86645614da1" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44561 } }
I20260926 16:49:00.597249 30564 leader_election.cc:288] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e6f27df3479439ea7ba80103602b144 (127.9.238.195:44497), 5981cbda4370475db447a8ece9df929f (127.9.238.194:44397)
I20260926 16:49:00.605675 30555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:00.597875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:37319 (local address 127.9.238.195:44497)
0926 16:49:00.598145 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:00.598154 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:00.599084 (+ 930us) server_negotiation.cc:407] Connection header received
0926 16:49:00.599493 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:00.599501 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:00.599640 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:00.599782 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:49:00.600815 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.601790 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.603846 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.604651 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.604843 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:00.604928 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:00.605202 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 16:49:00.605350 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 16:49:00.605706 30567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:00.597507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.195:44497 (local address 127.9.238.193:37319)
0926 16:49:00.598916 (+ 1409us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:00.598957 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 16:49:00.599224 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:49:00.600041 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:49:00.600047 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:49:00.600085 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:49:00.600595 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:00.600611 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.601937 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.601946 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:00.603634 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:00.603655 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.604856 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.604862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:00.605035 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:00.605062 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 16:49:00.605175 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 16:49:00.605335 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1251,"thread_start_us":1081,"threads_started":1}
I20260926 16:49:00.606061 30554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:00.598796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.9.238.193:54143 (local address 127.9.238.194:44397)
0926 16:49:00.599198 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:00.599206 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:00.600374 (+ 1168us) server_negotiation.cc:407] Connection header received
0926 16:49:00.600685 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:00.600691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:00.600793 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:00.600861 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 16:49:00.601752 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.602433 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.604338 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.605098 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.605282 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:00.605355 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:00.605618 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 16:49:00.605741 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260926 16:49:00.606091 30568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:00.599157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.194:44397 (local address 127.9.238.193:54143)
0926 16:49:00.600241 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:00.600260 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 16:49:00.600504 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:49:00.601131 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:49:00.601135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:49:00.601174 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 16:49:00.601580 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:00.601595 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.602579 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.602584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:00.604162 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:00.604175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.605299 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.605306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:00.605464 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:00.605489 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 16:49:00.605590 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 16:49:00.605738 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"thread_start_us":582,"threads_started":1}
I20260926 16:49:00.607090 30494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9347a6b0031d4a5ba34e2fea3ade2ce0" candidate_uuid: "78ead6c10c80474487ada86645614da1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e6f27df3479439ea7ba80103602b144" is_pre_election: true
I20260926 16:49:00.607363 30364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9347a6b0031d4a5ba34e2fea3ade2ce0" candidate_uuid: "78ead6c10c80474487ada86645614da1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5981cbda4370475db447a8ece9df929f" is_pre_election: true
I20260926 16:49:00.607679 30494 raft_consensus.cc:2454] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 6e6f27df3479439ea7ba80103602b144 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78ead6c10c80474487ada86645614da1 in term 0.
I20260926 16:49:00.608155 30364 raft_consensus.cc:2454] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 5981cbda4370475db447a8ece9df929f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78ead6c10c80474487ada86645614da1 in term 0.
I20260926 16:49:00.608421 30167 leader_election.cc:302] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e6f27df3479439ea7ba80103602b144, 78ead6c10c80474487ada86645614da1; no voters:
I20260926 16:49:00.608878 30564 raft_consensus.cc:2781] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 16:49:00.609018 30564 raft_consensus.cc:482] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 16:49:00.609191 30564 raft_consensus.cc:3037] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:49:00.611479 30564 raft_consensus.cc:504] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6f27df3479439ea7ba80103602b144" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44497 } } peers { permanent_uuid: "5981cbda4370475db447a8ece9df929f" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44397 } } peers { permanent_uuid: "78ead6c10c80474487ada86645614da1" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44561 } }
I20260926 16:49:00.612288 30564 leader_election.cc:288] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1 [CANDIDATE]: Term 1 election: Requested vote from peers 6e6f27df3479439ea7ba80103602b144 (127.9.238.195:44497), 5981cbda4370475db447a8ece9df929f (127.9.238.194:44397)
I20260926 16:49:00.612885 30494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9347a6b0031d4a5ba34e2fea3ade2ce0" candidate_uuid: "78ead6c10c80474487ada86645614da1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e6f27df3479439ea7ba80103602b144"
I20260926 16:49:00.613108 30364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9347a6b0031d4a5ba34e2fea3ade2ce0" candidate_uuid: "78ead6c10c80474487ada86645614da1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5981cbda4370475db447a8ece9df929f"
I20260926 16:49:00.613173 30494 raft_consensus.cc:3037] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 6e6f27df3479439ea7ba80103602b144 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:49:00.613422 30364 raft_consensus.cc:3037] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 5981cbda4370475db447a8ece9df929f [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:49:00.616168 30494 raft_consensus.cc:2454] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 6e6f27df3479439ea7ba80103602b144 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78ead6c10c80474487ada86645614da1 in term 1.
I20260926 16:49:00.616200 30364 raft_consensus.cc:2454] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 5981cbda4370475db447a8ece9df929f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78ead6c10c80474487ada86645614da1 in term 1.
I20260926 16:49:00.616681 30167 leader_election.cc:302] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e6f27df3479439ea7ba80103602b144, 78ead6c10c80474487ada86645614da1; no voters:
I20260926 16:49:00.617044 30564 raft_consensus.cc:2781] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:49:00.617628 30564 raft_consensus.cc:686] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1 [term 1 LEADER]: Becoming Leader. State: Replica: 78ead6c10c80474487ada86645614da1, State: Running, Role: LEADER
I20260926 16:49:00.618148 30564 consensus_queue.cc:237] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6f27df3479439ea7ba80103602b144" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44497 } } peers { permanent_uuid: "5981cbda4370475db447a8ece9df929f" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44397 } } peers { permanent_uuid: "78ead6c10c80474487ada86645614da1" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44561 } }
I20260926 16:49:00.624099 30090 catalog_manager.cc:5495] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78ead6c10c80474487ada86645614da1 (127.9.238.193). New cstate: current_term: 1 leader_uuid: "78ead6c10c80474487ada86645614da1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e6f27df3479439ea7ba80103602b144" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5981cbda4370475db447a8ece9df929f" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78ead6c10c80474487ada86645614da1" member_type: VOTER last_known_addr { host: "127.9.238.193" port: 44561 } health_report { overall_health: HEALTHY } } }
I20260926 16:49:00.678519 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:00.671877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57494 (local address 127.9.238.193:44561)
0926 16:49:00.672121 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:00.672126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:00.672462 (+ 336us) server_negotiation.cc:407] Connection header received
0926 16:49:00.672621 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:00.672626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:00.672748 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:00.672952 (+ 204us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:49:00.673435 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.674229 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.675836 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.676396 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.676522 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:00.677061 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:49:00.677861 (+ 800us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:49:00.678017 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:00.678118 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 16:49:00.678203 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 16:49:00.688758 30494 raft_consensus.cc:1261] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 6e6f27df3479439ea7ba80103602b144 [term 1 FOLLOWER]: Refusing update from remote peer 78ead6c10c80474487ada86645614da1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:49:00.688758 30364 raft_consensus.cc:1261] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 5981cbda4370475db447a8ece9df929f [term 1 FOLLOWER]: Refusing update from remote peer 78ead6c10c80474487ada86645614da1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:49:00.690193 30564 consensus_queue.cc:1035] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e6f27df3479439ea7ba80103602b144" member_type: VOTER last_known_addr { host: "127.9.238.195" port: 44497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:49:00.690689 30569 consensus_queue.cc:1035] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5981cbda4370475db447a8ece9df929f" member_type: VOTER last_known_addr { host: "127.9.238.194" port: 44397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:49:00.704077 30573 mvcc.cc:205] Tried to move back new op lower bound from 7333647731448233984 to 7333647731175628800. Current Snapshot: MvccSnapshot[applied={T|T < 7333647731448233984}]
I20260926 16:49:00.708811 30574 mvcc.cc:205] Tried to move back new op lower bound from 7333647731448233984 to 7333647731175628800. Current Snapshot: MvccSnapshot[applied={T|T < 7333647731448233984}]
I20260926 16:49:00.723112 30091 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:49:00.723506 30091 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 16:49:00.728547 30091 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95a5467ad64349a280ae0fcba7ab02dc: Sending DeleteTablet for 3 replicas of tablet 9347a6b0031d4a5ba34e2fea3ade2ce0
I20260926 16:49:00.730007 30213 tablet_service.cc:1514] Processing DeleteTablet for tablet 9347a6b0031d4a5ba34e2fea3ade2ce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:49:00 UTC) from {username='slave'} at 127.0.0.1:57478
I20260926 16:49:00.730156 30344 tablet_service.cc:1514] Processing DeleteTablet for tablet 9347a6b0031d4a5ba34e2fea3ade2ce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:49:00 UTC) from {username='slave'} at 127.0.0.1:51274
I20260926 16:49:00.730157 30474 tablet_service.cc:1514] Processing DeleteTablet for tablet 9347a6b0031d4a5ba34e2fea3ade2ce0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 16:49:00 UTC) from {username='slave'} at 127.0.0.1:40460
I20260926 16:49:00.732116 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 30151
I20260926 16:49:00.748832 30582 tablet_replica.cc:330] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 6e6f27df3479439ea7ba80103602b144: stopping tablet replica
I20260926 16:49:00.749665 30582 raft_consensus.cc:2229] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 6e6f27df3479439ea7ba80103602b144 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:49:00.750953 30584 tablet_replica.cc:330] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1: stopping tablet replica
I20260926 16:49:00.751348 30582 raft_consensus.cc:2258] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 6e6f27df3479439ea7ba80103602b144 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:49:00.752509 30583 tablet_replica.cc:330] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 5981cbda4370475db447a8ece9df929f: stopping tablet replica
I20260926 16:49:00.753002 30584 raft_consensus.cc:2229] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1 [term 1 LEADER]: Raft consensus shutting down.
I20260926 16:49:00.753856 30557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:00.734181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57500 (local address 127.9.238.193:44561)
0926 16:49:00.734486 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:00.734496 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:00.739257 (+ 4761us) server_negotiation.cc:407] Connection header received
0926 16:49:00.740437 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:00.740448 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:00.740651 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:00.740767 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:00.743126 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.744870 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.748544 (+ 3674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.749344 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.749481 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:00.751678 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:00.751692 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:00.751717 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:00.752747 (+ 1030us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:00.752821 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:00.752851 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:00.752984 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:00.753224 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 16:49:00.753371 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 16:49:00.754190 30583 raft_consensus.cc:2229] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 5981cbda4370475db447a8ece9df929f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 16:49:00.754735 30583 raft_consensus.cc:2258] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 5981cbda4370475db447a8ece9df929f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:49:00.754813 30582 ts_tablet_manager.cc:1905] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 6e6f27df3479439ea7ba80103602b144: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:49:00.755400 30584 raft_consensus.cc:2258] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 16:49:00.755522 30273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:49:00.757170 30583 ts_tablet_manager.cc:1905] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 5981cbda4370475db447a8ece9df929f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 16:49:00.763314 30582 ts_tablet_manager.cc:1918] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 6e6f27df3479439ea7ba80103602b144: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:49:00.763669 30582 log.cc:1196] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 6e6f27df3479439ea7ba80103602b144: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-2/wal/wals/9347a6b0031d4a5ba34e2fea3ade2ce0
I20260926 16:49:00.764565 30582 ts_tablet_manager.cc:1939] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 6e6f27df3479439ea7ba80103602b144: Deleting consensus metadata
I20260926 16:49:00.765388 30583 ts_tablet_manager.cc:1918] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 5981cbda4370475db447a8ece9df929f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 16:49:00.765709 30583 log.cc:1196] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 5981cbda4370475db447a8ece9df929f: Deleting WAL directory at /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790440988577176-10171-0/minicluster-data/ts-1/wal/wals/9347a6b0031d4a5ba34e2fea3ade2ce0
I20260926 16:49:00.766359 30583 ts_tablet_manager.cc:1939] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 5981cbda4370475db447a8ece9df929f: Deleting consensus metadata
I20260926 16:49:00.767042 30077 catalog_manager.cc:4841] TS 6e6f27df3479439ea7ba80103602b144 (127.9.238.195:44497): tablet 9347a6b0031d4a5ba34e2fea3ade2ce0 (table test-table [id=139606a5fbe24586b7305b816b9d2287]) successfully deleted
I20260926 16:49:00.768121 30079 catalog_manager.cc:4841] TS 5981cbda4370475db447a8ece9df929f (127.9.238.194:44397): tablet 9347a6b0031d4a5ba34e2fea3ade2ce0 (table test-table [id=139606a5fbe24586b7305b816b9d2287]) successfully deleted
I20260926 16:49:00.870178 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 30151
I20260926 16:49:00.870255 30584 ts_tablet_manager.cc:1905] T 9347a6b0031d4a5ba34e2fea3ade2ce0 P 78ead6c10c80474487ada86645614da1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 16:49:00.889874 30078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.9.238.193:44561 (error 108)
I20260926 16:49:00.890847 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 30282
I20260926 16:49:00.892683 30588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:00.891508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:44561 (local address 127.0.0.1:57512)
0926 16:49:00.892151 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:00.892417 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:44561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":293,"threads_started":1}
W20260926 16:49:00.893034 30078 catalog_manager.cc:4584] TS 78ead6c10c80474487ada86645614da1 (127.9.238.193:44561): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9347a6b0031d4a5ba34e2fea3ade2ce0: Network error: Client connection negotiation failed: client connection to 127.9.238.193:44561: connect: Connection refused (error 111)
I20260926 16:49:00.899508 30554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:00.891870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51280 (local address 127.9.238.194:44397)
0926 16:49:00.892165 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:00.892170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:00.892324 (+ 154us) server_negotiation.cc:407] Connection header received
0926 16:49:00.892571 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:00.892576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:00.892690 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:00.892837 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:00.893438 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.894236 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.895839 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:00.896339 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:00.896467 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:00.897789 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:00.897800 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:00.897820 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:00.898657 (+ 837us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:00.898718 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:00.898740 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:00.898857 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:00.899081 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 16:49:00.899194 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260926 16:49:00.900334 30404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:49:00.947279 30588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:00.946679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:44561 (local address 127.0.0.1:57524)
0926 16:49:00.946938 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:00.947102 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:44561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260926 16:49:00.948478 30588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:00.948040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:44561 (local address 127.0.0.1:57526)
0926 16:49:00.948224 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:00.948320 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:44561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 16:49:00.993150 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 30282
I20260926 16:49:01.009478 30588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:01.008721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:44561 (local address 127.0.0.1:57528)
0926 16:49:01.009062 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:01.009221 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:44561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260926 16:49:01.010866 30588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:01.010379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:44561 (local address 127.0.0.1:57538)
0926 16:49:01.010579 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:01.010684 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:44561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 16:49:01.012571 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 30412
I20260926 16:49:01.020814 30555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:01.013507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40472 (local address 127.9.238.195:44497)
0926 16:49:01.013761 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:01.013768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:01.013826 (+ 58us) server_negotiation.cc:407] Connection header received
0926 16:49:01.013902 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:01.013907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:01.014017 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:01.014115 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:01.014640 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:01.015499 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:01.017128 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:01.017623 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:01.017734 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:01.018937 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:01.018959 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:01.018979 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:01.019801 (+ 822us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:01.019861 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:01.019883 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:01.019982 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:01.020163 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 16:49:01.020282 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 16:49:01.021778 30534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:49:01.092065 30588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:01.091266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:44561 (local address 127.0.0.1:57908)
0926 16:49:01.091678 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:01.091833 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:44561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260926 16:49:01.093531 30588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:01.093023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.193:44561 (local address 127.0.0.1:57914)
0926 16:49:01.093268 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:01.093358 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.193:44561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260926 16:49:01.127420 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 30412
I20260926 16:49:01.142339 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 30061
I20260926 16:49:01.143886 30122 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:49:01.274260 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 30061
2026-09-26T16:49:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3449 ms)
[----------] 6 tests from EncryptionPolicyTest (24871 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:49:01 dist-test-slave-wt2j krb5kdc[30594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:49:01 dist-test-slave-wt2j krb5kdc[30594](info): set up 2 sockets
Sep 26 16:49:01 dist-test-slave-wt2j krb5kdc[30594](info): commencing operation
krb5kdc: starting...
W20260926 16:49:02.469555 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:49:02 dist-test-slave-wt2j krb5kdc[30594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:49:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:49:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:49:02.784348 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43761
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:49:03.154167 30607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:49:03.154553 30607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:49:03.154721 30607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:49:03.162249 30607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:49:03.162431 30607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:49:03.162539 30607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:49:03.162655 30607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:49:03.162742 30607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:49:03.172758 30607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43761
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:49:03.174166 30607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:49:03.175494 30607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:49:03.188459 30612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:03.189024 30613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:03.189643 30615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:49:03.190094 30607 server_base.cc:1034] running on GCE node
I20260926 16:49:03.190938 30607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:49:03.192557 30607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:49:03.193807 30607 hybrid_clock.cc:648] HybridClock initialized: now 1790441343193737 us; error 62 us; skew 500 ppm
I20260926 16:49:03.194263 30607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:49:03 dist-test-slave-wt2j krb5kdc[30594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:49:03.198129 30607 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:49:03.200028 30607 webserver.cc:460] Webserver started at http://127.0.0.1:38105/ using document root <none> and password file <none>
I20260926 16:49:03.200773 30607 fs_manager.cc:359] Metadata directory not provided
I20260926 16:49:03.200906 30607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:49:03.201248 30607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:49:03.204412 30607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "aeef8955318f4cbcb0f15ed264fe0c3f"
format_stamp: "Formatted at 2026-09-26 16:49:03 on dist-test-slave-wt2j"
server_key: "c20dca9c05a40929b4694534c2154831"
server_key_iv: "027ce79c46b589c1f002d694eced5759"
server_key_version: "encryptionkey@0"
I20260926 16:49:03.205215 30607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "aeef8955318f4cbcb0f15ed264fe0c3f"
format_stamp: "Formatted at 2026-09-26 16:49:03 on dist-test-slave-wt2j"
server_key: "c20dca9c05a40929b4694534c2154831"
server_key_iv: "027ce79c46b589c1f002d694eced5759"
server_key_version: "encryptionkey@0"
I20260926 16:49:03.210974 30607 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260926 16:49:03.215747 30622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:49:03.217612 30607 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:49:03.217844 30607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "aeef8955318f4cbcb0f15ed264fe0c3f"
format_stamp: "Formatted at 2026-09-26 16:49:03 on dist-test-slave-wt2j"
server_key: "c20dca9c05a40929b4694534c2154831"
server_key_iv: "027ce79c46b589c1f002d694eced5759"
server_key_version: "encryptionkey@0"
I20260926 16:49:03.218139 30607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:49:03.250324 30607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:49:03.256418 30607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:49:03.256815 30607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:49:03.280608 30607 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43761
I20260926 16:49:03.280665 30673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43761 every 8 connection(s)
I20260926 16:49:03.282555 30607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:49:03.287232 30674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:49:03.287325 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 30607
I20260926 16:49:03.287588 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:49:03.288514 10171 external_mini_cluster.cc:1428] Setting key e827e0b62f8e23039e436f1ee83f621b
I20260926 16:49:03.304062 30674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeef8955318f4cbcb0f15ed264fe0c3f: Bootstrap starting.
Sep 26 16:49:03 dist-test-slave-wt2j krb5kdc[30594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 16:49:03.307422 30674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aeef8955318f4cbcb0f15ed264fe0c3f: Neither blocks nor log segments found. Creating new log.
I20260926 16:49:03.308523 30674 log.cc:824] T 00000000000000000000000000000000 P aeef8955318f4cbcb0f15ed264fe0c3f: Log is configured to *not* fsync() on all Append() calls
I20260926 16:49:03.311785 30674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeef8955318f4cbcb0f15ed264fe0c3f: No bootstrap required, opened a new log
I20260926 16:49:03.318082 30677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:03.291148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48865 (local address 127.0.0.1:43761)
0926 16:49:03.292665 (+ 1517us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:03.292693 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:03.292772 (+ 79us) server_negotiation.cc:407] Connection header received
0926 16:49:03.295124 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:03.295162 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:03.295987 (+ 825us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:03.296520 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:03.298470 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:03.300235 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:03.302248 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:03.303142 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:03.303312 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:03.308325 (+ 5013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:03.308347 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:03.308370 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:03.308433 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:03.312536 (+ 4103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:03.313774 (+ 1238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:03.313783 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:03.313791 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:03.314031 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:03.314557 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:03.314566 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:03.314568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:03.315295 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:03.315719 (+ 424us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:03.316294 (+ 575us) server_negotiation.cc:299] Negotiation successful
0926 16:49:03.316722 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":926,"thread_start_us":500,"threads_started":1}
I20260926 16:49:03.320640 30674 raft_consensus.cc:348] T 00000000000000000000000000000000 P aeef8955318f4cbcb0f15ed264fe0c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeef8955318f4cbcb0f15ed264fe0c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43761 } }
I20260926 16:49:03.321122 30674 raft_consensus.cc:374] T 00000000000000000000000000000000 P aeef8955318f4cbcb0f15ed264fe0c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:49:03.321233 30674 raft_consensus.cc:729] T 00000000000000000000000000000000 P aeef8955318f4cbcb0f15ed264fe0c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeef8955318f4cbcb0f15ed264fe0c3f, State: Initialized, Role: FOLLOWER
I20260926 16:49:03.322127 30674 consensus_queue.cc:260] T 00000000000000000000000000000000 P aeef8955318f4cbcb0f15ed264fe0c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeef8955318f4cbcb0f15ed264fe0c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43761 } }
I20260926 16:49:03.322446 30674 raft_consensus.cc:388] T 00000000000000000000000000000000 P aeef8955318f4cbcb0f15ed264fe0c3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:49:03.322593 30674 raft_consensus.cc:482] T 00000000000000000000000000000000 P aeef8955318f4cbcb0f15ed264fe0c3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:49:03.322813 30674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aeef8955318f4cbcb0f15ed264fe0c3f [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:49:03.323837 10171 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:49:03.326146 30674 raft_consensus.cc:504] T 00000000000000000000000000000000 P aeef8955318f4cbcb0f15ed264fe0c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeef8955318f4cbcb0f15ed264fe0c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43761 } }
I20260926 16:49:03.326802 30674 leader_election.cc:302] T 00000000000000000000000000000000 P aeef8955318f4cbcb0f15ed264fe0c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aeef8955318f4cbcb0f15ed264fe0c3f; no voters:
I20260926 16:49:03.327694 30674 leader_election.cc:288] T 00000000000000000000000000000000 P aeef8955318f4cbcb0f15ed264fe0c3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:49:03.328037 30679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aeef8955318f4cbcb0f15ed264fe0c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:49:03.329325 30679 raft_consensus.cc:686] T 00000000000000000000000000000000 P aeef8955318f4cbcb0f15ed264fe0c3f [term 1 LEADER]: Becoming Leader. State: Replica: aeef8955318f4cbcb0f15ed264fe0c3f, State: Running, Role: LEADER
I20260926 16:49:03.330169 30679 consensus_queue.cc:237] T 00000000000000000000000000000000 P aeef8955318f4cbcb0f15ed264fe0c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeef8955318f4cbcb0f15ed264fe0c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43761 } }
I20260926 16:49:03.330777 30674 sys_catalog.cc:564] T 00000000000000000000000000000000 P aeef8955318f4cbcb0f15ed264fe0c3f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:49:03.341483 30682 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeef8955318f4cbcb0f15ed264fe0c3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aeef8955318f4cbcb0f15ed264fe0c3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeef8955318f4cbcb0f15ed264fe0c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43761 } } }
I20260926 16:49:03.342042 30682 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeef8955318f4cbcb0f15ed264fe0c3f [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:03.342687 30684 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeef8955318f4cbcb0f15ed264fe0c3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader aeef8955318f4cbcb0f15ed264fe0c3f. Latest consensus state: current_term: 1 leader_uuid: "aeef8955318f4cbcb0f15ed264fe0c3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aeef8955318f4cbcb0f15ed264fe0c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43761 } } }
I20260926 16:49:03.343003 30684 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeef8955318f4cbcb0f15ed264fe0c3f [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:03.346259 30694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:49:03.353765 30694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:49:03.358922 30677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:03.332659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40029 (local address 127.0.0.1:43761)
0926 16:49:03.333087 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:03.333097 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:03.335512 (+ 2415us) server_negotiation.cc:407] Connection header received
0926 16:49:03.335885 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:03.335895 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:03.336045 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:03.336226 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:03.339101 (+ 2875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:03.340774 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:03.343287 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:03.344410 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:03.344550 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:03.348813 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:03.348821 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:03.348826 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:03.348881 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:03.352893 (+ 4012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:03.353857 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:03.353866 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:03.353872 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:03.354067 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:03.354786 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:03.354795 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:03.354801 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:03.355449 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:03.355864 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:03.357918 (+ 2054us) server_negotiation.cc:299] Negotiation successful
0926 16:49:03.358246 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242}
I20260926 16:49:03.365137 30694 catalog_manager.cc:1347] Generated new cluster ID: cc9a381e12bb4f1eab6de40cb1045142
I20260926 16:49:03.365288 30694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:49:03.379593 30694 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:49:03.381194 30694 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:49:03.398236 30694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aeef8955318f4cbcb0f15ed264fe0c3f: Generated new TSK 0
I20260926 16:49:03.399500 30694 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:49:03.410682 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 30607
I20260926 16:49:03.412627 30669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:49:03.518241 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 30607
W20260926 16:49:03.539980 30592 connection.cc:535] client connection to 127.0.0.1:43761 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43761): Connection reset by peer (error 104)
2026-09-26T16:49:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2234 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:49:03 dist-test-slave-wt2j krb5kdc[30706](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:49:03 dist-test-slave-wt2j krb5kdc[30706](info): set up 2 sockets
Sep 26 16:49:03 dist-test-slave-wt2j krb5kdc[30706](info): commencing operation
krb5kdc: starting...
W20260926 16:49:04.685886 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.004s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:49:04 dist-test-slave-wt2j krb5kdc[30706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:49:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:49:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:49:05.000420 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36343
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:49:05.372421 30719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:49:05.372732 30719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:49:05.372825 30719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:49:05.380095 30719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:49:05.380223 30719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:49:05.380268 30719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:49:05.380378 30719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:49:05.380455 30719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:49:05.390610 30719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36343
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:49:05.392089 30719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:49:05.393421 30719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:49:05.407188 30725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:05.407148 30724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:05.407246 30727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:49:05.408372 30719 server_base.cc:1034] running on GCE node
I20260926 16:49:05.409359 30719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:49:05.411280 30719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:49:05.412552 30719 hybrid_clock.cc:648] HybridClock initialized: now 1790441345412473 us; error 67 us; skew 500 ppm
I20260926 16:49:05.412951 30719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:49:05 dist-test-slave-wt2j krb5kdc[30706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:49:05.421667 30719 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:49:05.423780 30719 webserver.cc:460] Webserver started at http://127.0.0.1:38267/ using document root <none> and password file <none>
I20260926 16:49:05.424693 30719 fs_manager.cc:359] Metadata directory not provided
I20260926 16:49:05.424858 30719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:49:05.425238 30719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:49:05.429185 30719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "274b47b5a6b14593a2e0f21f5fc01cb0"
format_stamp: "Formatted at 2026-09-26 16:49:05 on dist-test-slave-wt2j"
server_key: "a10f1985ef1dcf28f7ed0b5a4bff1f2d"
server_key_iv: "7f9ca1db0e5cdce7c50407b68568dcca"
server_key_version: "encryptionkey@0"
I20260926 16:49:05.430502 30719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "274b47b5a6b14593a2e0f21f5fc01cb0"
format_stamp: "Formatted at 2026-09-26 16:49:05 on dist-test-slave-wt2j"
server_key: "a10f1985ef1dcf28f7ed0b5a4bff1f2d"
server_key_iv: "7f9ca1db0e5cdce7c50407b68568dcca"
server_key_version: "encryptionkey@0"
I20260926 16:49:05.437978 30719 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260926 16:49:05.443698 30734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:49:05.445654 30719 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 16:49:05.445924 30719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "274b47b5a6b14593a2e0f21f5fc01cb0"
format_stamp: "Formatted at 2026-09-26 16:49:05 on dist-test-slave-wt2j"
server_key: "a10f1985ef1dcf28f7ed0b5a4bff1f2d"
server_key_iv: "7f9ca1db0e5cdce7c50407b68568dcca"
server_key_version: "encryptionkey@0"
I20260926 16:49:05.446197 30719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:49:05.482110 30719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:49:05.488332 30719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:49:05.488723 30719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:49:05.513702 30719 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36343
I20260926 16:49:05.513739 30785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36343 every 8 connection(s)
I20260926 16:49:05.515858 30719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:49:05.520498 30786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:49:05.523972 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 30719
I20260926 16:49:05.524180 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:49:05.524719 10171 external_mini_cluster.cc:1428] Setting key 8b2533afc537e502ddc7217061d53507
I20260926 16:49:05.535678 30786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 274b47b5a6b14593a2e0f21f5fc01cb0: Bootstrap starting.
I20260926 16:49:05.539515 30786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 274b47b5a6b14593a2e0f21f5fc01cb0: Neither blocks nor log segments found. Creating new log.
I20260926 16:49:05.541009 30786 log.cc:824] T 00000000000000000000000000000000 P 274b47b5a6b14593a2e0f21f5fc01cb0: Log is configured to *not* fsync() on all Append() calls
I20260926 16:49:05.542649 30789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:05.527157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46635 (local address 127.0.0.1:36343)
0926 16:49:05.528628 (+ 1471us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:05.528649 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:05.528711 (+ 62us) server_negotiation.cc:407] Connection header received
0926 16:49:05.530907 (+ 2196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:05.530943 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:05.531523 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:05.532059 (+ 536us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:05.532749 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:05.534779 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:05.536647 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:05.537765 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:05.537976 (+ 211us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:05.539482 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:05.539499 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:05.539520 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:05.540365 (+ 845us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:05.540476 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:05.540510 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:05.540723 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:05.541014 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 16:49:05.541208 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1166,"thread_start_us":889,"threads_started":1}
I20260926 16:49:05.545924 30786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 274b47b5a6b14593a2e0f21f5fc01cb0: No bootstrap required, opened a new log
I20260926 16:49:05.548105 10171 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:49:05.556778 30786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 274b47b5a6b14593a2e0f21f5fc01cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274b47b5a6b14593a2e0f21f5fc01cb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36343 } }
I20260926 16:49:05.557394 30786 raft_consensus.cc:374] T 00000000000000000000000000000000 P 274b47b5a6b14593a2e0f21f5fc01cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:49:05.557548 30786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 274b47b5a6b14593a2e0f21f5fc01cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274b47b5a6b14593a2e0f21f5fc01cb0, State: Initialized, Role: FOLLOWER
I20260926 16:49:05.558423 30786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 274b47b5a6b14593a2e0f21f5fc01cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274b47b5a6b14593a2e0f21f5fc01cb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36343 } }
I20260926 16:49:05.558768 30786 raft_consensus.cc:388] T 00000000000000000000000000000000 P 274b47b5a6b14593a2e0f21f5fc01cb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:49:05.558980 30786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 274b47b5a6b14593a2e0f21f5fc01cb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:49:05.559255 30786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 274b47b5a6b14593a2e0f21f5fc01cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:49:05.561043 30789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:05.552998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49987 (local address 127.0.0.1:36343)
0926 16:49:05.553275 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:05.553283 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:05.553655 (+ 372us) server_negotiation.cc:407] Connection header received
0926 16:49:05.553830 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:05.553841 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:05.553959 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:05.554127 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:05.554635 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:05.555985 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:05.557591 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:05.558277 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:05.558427 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:05.559921 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:05.559932 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:05.559937 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:05.559967 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:05.560058 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:05.560087 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:05.560259 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:05.560507 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 16:49:05.560593 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260926 16:49:05.563241 30786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 274b47b5a6b14593a2e0f21f5fc01cb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274b47b5a6b14593a2e0f21f5fc01cb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36343 } }
I20260926 16:49:05.563956 30786 leader_election.cc:302] T 00000000000000000000000000000000 P 274b47b5a6b14593a2e0f21f5fc01cb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 274b47b5a6b14593a2e0f21f5fc01cb0; no voters:
I20260926 16:49:05.564678 30786 leader_election.cc:288] T 00000000000000000000000000000000 P 274b47b5a6b14593a2e0f21f5fc01cb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:49:05.564905 30796 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 274b47b5a6b14593a2e0f21f5fc01cb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:49:05.566316 30796 raft_consensus.cc:686] T 00000000000000000000000000000000 P 274b47b5a6b14593a2e0f21f5fc01cb0 [term 1 LEADER]: Becoming Leader. State: Replica: 274b47b5a6b14593a2e0f21f5fc01cb0, State: Running, Role: LEADER
I20260926 16:49:05.566988 30796 consensus_queue.cc:237] T 00000000000000000000000000000000 P 274b47b5a6b14593a2e0f21f5fc01cb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274b47b5a6b14593a2e0f21f5fc01cb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36343 } }
I20260926 16:49:05.567787 30786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 274b47b5a6b14593a2e0f21f5fc01cb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:49:05.570785 30798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 274b47b5a6b14593a2e0f21f5fc01cb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 274b47b5a6b14593a2e0f21f5fc01cb0. Latest consensus state: current_term: 1 leader_uuid: "274b47b5a6b14593a2e0f21f5fc01cb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274b47b5a6b14593a2e0f21f5fc01cb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36343 } } }
I20260926 16:49:05.571276 30798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 274b47b5a6b14593a2e0f21f5fc01cb0 [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:05.571763 30797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 274b47b5a6b14593a2e0f21f5fc01cb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "274b47b5a6b14593a2e0f21f5fc01cb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "274b47b5a6b14593a2e0f21f5fc01cb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36343 } } }
I20260926 16:49:05.572153 30797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 274b47b5a6b14593a2e0f21f5fc01cb0 [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:05.572679 30805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:49:05.581889 30805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:49:05.594072 30805 catalog_manager.cc:1347] Generated new cluster ID: c6d43c2a817b4e28a2f1eabe82ac8225
I20260926 16:49:05.594244 30805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:49:05.608271 30805 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:49:05.610471 30805 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:49:05.627091 30805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 274b47b5a6b14593a2e0f21f5fc01cb0: Generated new TSK 0
I20260926 16:49:05.628401 30805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:49:05.646081 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 30719
W20260926 16:49:05.647886 30781 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46635
W20260926 16:49:05.648921 10171 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 30719: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:49:05.649150 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 30719
W20260926 16:49:05.672739 30704 connection.cc:535] client connection to 127.0.0.1:36343 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36343): Connection reset by peer (error 104)
2026-09-26T16:49:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2132 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:49:05 dist-test-slave-wt2j krb5kdc[30817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:49:05 dist-test-slave-wt2j krb5kdc[30817](info): set up 2 sockets
Sep 26 16:49:05 dist-test-slave-wt2j krb5kdc[30817](info): commencing operation
krb5kdc: starting...
W20260926 16:49:06.853688 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.003s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:49:07 dist-test-slave-wt2j krb5kdc[30817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:49:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:49:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:49:07.171958 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41421
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:49:07.529042 30830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:49:07.529383 30830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:49:07.529481 30830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:49:07.537115 30830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:49:07.537261 30830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:49:07.537309 30830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:49:07.537364 30830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:49:07.537415 30830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:49:07.548804 30830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41421
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:49:07.550140 30830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:49:07.551446 30830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:49:07.565425 30835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:07.565384 30838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:07.565282 30836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:49:07.566751 30830 server_base.cc:1034] running on GCE node
I20260926 16:49:07.567808 30830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:49:07.569499 30830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:49:07.570776 30830 hybrid_clock.cc:648] HybridClock initialized: now 1790441347570725 us; error 75 us; skew 500 ppm
I20260926 16:49:07.571179 30830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:49:07 dist-test-slave-wt2j krb5kdc[30817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:49:07.575306 30830 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:49:07.577280 30830 webserver.cc:460] Webserver started at http://127.0.0.1:42647/ using document root <none> and password file <none>
I20260926 16:49:07.578171 30830 fs_manager.cc:359] Metadata directory not provided
I20260926 16:49:07.578293 30830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:49:07.578686 30830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:49:07.582028 30830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "93700ba52e5a44d5a41265f900241756"
format_stamp: "Formatted at 2026-09-26 16:49:07 on dist-test-slave-wt2j"
server_key: "b34502b594ac42a4dac5cee104c821ea"
server_key_iv: "46dca8c1fcd2566dba6ae02e1aeb88a2"
server_key_version: "encryptionkey@0"
I20260926 16:49:07.582894 30830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "93700ba52e5a44d5a41265f900241756"
format_stamp: "Formatted at 2026-09-26 16:49:07 on dist-test-slave-wt2j"
server_key: "b34502b594ac42a4dac5cee104c821ea"
server_key_iv: "46dca8c1fcd2566dba6ae02e1aeb88a2"
server_key_version: "encryptionkey@0"
I20260926 16:49:07.589228 30830 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.006s
I20260926 16:49:07.593784 30845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:49:07.595925 30830 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 16:49:07.596184 30830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "93700ba52e5a44d5a41265f900241756"
format_stamp: "Formatted at 2026-09-26 16:49:07 on dist-test-slave-wt2j"
server_key: "b34502b594ac42a4dac5cee104c821ea"
server_key_iv: "46dca8c1fcd2566dba6ae02e1aeb88a2"
server_key_version: "encryptionkey@0"
I20260926 16:49:07.596428 30830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:49:07.603176 30830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:49:07.608105 30830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:49:07.608413 30830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:49:07.629773 30830 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41421
I20260926 16:49:07.629876 30896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41421 every 8 connection(s)
I20260926 16:49:07.631508 30830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:49:07.634828 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 30830
I20260926 16:49:07.635285 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:49:07.636049 10171 external_mini_cluster.cc:1428] Setting key 996f289fbe86688ef0efe4cb2ee20bc0
I20260926 16:49:07.638185 30897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:49:07.645462 30900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:07.638345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49207 (local address 127.0.0.1:41421)
0926 16:49:07.639446 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:07.639462 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:07.639640 (+ 178us) server_negotiation.cc:407] Connection header received
0926 16:49:07.641477 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:07.641504 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:07.641951 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:07.642395 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:07.643622 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:07.643634 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:07.643648 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:07.643664 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:07.643754 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:07.643774 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:07.644044 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:07.644190 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 16:49:07.644408 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":805,"thread_start_us":476,"threads_started":1}
I20260926 16:49:07.649672 10171 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:49:07.653510 30897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93700ba52e5a44d5a41265f900241756: Bootstrap starting.
I20260926 16:49:07.657876 30900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:07.654617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54861 (local address 127.0.0.1:41421)
0926 16:49:07.654844 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:07.654850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:07.655321 (+ 471us) server_negotiation.cc:407] Connection header received
0926 16:49:07.655670 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:07.655675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:07.655746 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:07.655877 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:07.657071 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:07.657076 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:07.657079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:07.657094 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:07.657135 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:07.657151 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:07.657256 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:07.657460 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 16:49:07.657528 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 16:49:07.658771 30897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93700ba52e5a44d5a41265f900241756: Neither blocks nor log segments found. Creating new log.
I20260926 16:49:07.660564 30897 log.cc:824] T 00000000000000000000000000000000 P 93700ba52e5a44d5a41265f900241756: Log is configured to *not* fsync() on all Append() calls
I20260926 16:49:07.664942 30897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93700ba52e5a44d5a41265f900241756: No bootstrap required, opened a new log
I20260926 16:49:07.674612 30897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93700ba52e5a44d5a41265f900241756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93700ba52e5a44d5a41265f900241756" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41421 } }
I20260926 16:49:07.675146 30897 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93700ba52e5a44d5a41265f900241756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:49:07.675303 30897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93700ba52e5a44d5a41265f900241756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93700ba52e5a44d5a41265f900241756, State: Initialized, Role: FOLLOWER
I20260926 16:49:07.676198 30897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93700ba52e5a44d5a41265f900241756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93700ba52e5a44d5a41265f900241756" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41421 } }
I20260926 16:49:07.676661 30897 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93700ba52e5a44d5a41265f900241756 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:49:07.676877 30897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93700ba52e5a44d5a41265f900241756 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:49:07.677086 30897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93700ba52e5a44d5a41265f900241756 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:49:07.679838 30897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93700ba52e5a44d5a41265f900241756 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93700ba52e5a44d5a41265f900241756" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41421 } }
I20260926 16:49:07.680570 30897 leader_election.cc:302] T 00000000000000000000000000000000 P 93700ba52e5a44d5a41265f900241756 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93700ba52e5a44d5a41265f900241756; no voters:
I20260926 16:49:07.681387 30897 leader_election.cc:288] T 00000000000000000000000000000000 P 93700ba52e5a44d5a41265f900241756 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:49:07.681689 30907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93700ba52e5a44d5a41265f900241756 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:49:07.682919 30907 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93700ba52e5a44d5a41265f900241756 [term 1 LEADER]: Becoming Leader. State: Replica: 93700ba52e5a44d5a41265f900241756, State: Running, Role: LEADER
I20260926 16:49:07.683640 30907 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93700ba52e5a44d5a41265f900241756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93700ba52e5a44d5a41265f900241756" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41421 } }
I20260926 16:49:07.684082 30897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93700ba52e5a44d5a41265f900241756 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:49:07.686837 30909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93700ba52e5a44d5a41265f900241756 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93700ba52e5a44d5a41265f900241756. Latest consensus state: current_term: 1 leader_uuid: "93700ba52e5a44d5a41265f900241756" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93700ba52e5a44d5a41265f900241756" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41421 } } }
I20260926 16:49:07.688136 30909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93700ba52e5a44d5a41265f900241756 [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:07.686852 30908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93700ba52e5a44d5a41265f900241756 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93700ba52e5a44d5a41265f900241756" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93700ba52e5a44d5a41265f900241756" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41421 } } }
I20260926 16:49:07.688293 30908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93700ba52e5a44d5a41265f900241756 [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:07.692399 30914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:49:07.701365 30914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:49:07.714095 30914 catalog_manager.cc:1347] Generated new cluster ID: e76537fa99884b20adf4680b8a39f12f
I20260926 16:49:07.714279 30914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:49:07.742113 30914 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:49:07.742774 30914 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:49:07.758152 30914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93700ba52e5a44d5a41265f900241756: Generated new TSK 0
I20260926 16:49:07.759420 30914 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:49:07.769047 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 30830
W20260926 16:49:07.770381 30892 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49207
W20260926 16:49:07.771534 10171 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 30830: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:49:07.771754 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 30830
2026-09-26T16:49:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2122 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:49:07 dist-test-slave-wt2j krb5kdc[30928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:49:07 dist-test-slave-wt2j krb5kdc[30928](info): set up 2 sockets
Sep 26 16:49:07 dist-test-slave-wt2j krb5kdc[30928](info): commencing operation
krb5kdc: starting...
W20260926 16:49:08.925433 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:49:09 dist-test-slave-wt2j krb5kdc[30928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:49:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:49:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:49:09.240917 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35877
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:49:09.583739 30941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:49:09.584115 30941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:49:09.584290 30941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:49:09.592952 30941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:49:09.593104 30941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:49:09.593240 30941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:49:09.593374 30941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:49:09.593478 30941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:49:09.604619 30941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:49:09.606050 30941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:49:09.607450 30941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:49:09.621790 30949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:09.621863 30947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:09.622160 30946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:49:09.623215 30941 server_base.cc:1034] running on GCE node
I20260926 16:49:09.624248 30941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:49:09.625939 30941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:49:09.627457 30941 hybrid_clock.cc:648] HybridClock initialized: now 1790441349627382 us; error 71 us; skew 500 ppm
I20260926 16:49:09.627960 30941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:49:09 dist-test-slave-wt2j krb5kdc[30928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:49:09.631937 30941 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:49:09.633934 30941 webserver.cc:460] Webserver started at http://127.0.0.1:40287/ using document root <none> and password file <none>
I20260926 16:49:09.634707 30941 fs_manager.cc:359] Metadata directory not provided
I20260926 16:49:09.634819 30941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:49:09.635198 30941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:49:09.638571 30941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "16f7b2e15c444168a19f174f49ae81bc"
format_stamp: "Formatted at 2026-09-26 16:49:09 on dist-test-slave-wt2j"
server_key: "20825d3f9ef46516afb251bdd30b10d3"
server_key_iv: "069414b52f33af39b8ce182fd5d27451"
server_key_version: "encryptionkey@0"
I20260926 16:49:09.639454 30941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "16f7b2e15c444168a19f174f49ae81bc"
format_stamp: "Formatted at 2026-09-26 16:49:09 on dist-test-slave-wt2j"
server_key: "20825d3f9ef46516afb251bdd30b10d3"
server_key_iv: "069414b52f33af39b8ce182fd5d27451"
server_key_version: "encryptionkey@0"
I20260926 16:49:09.647176 30941 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260926 16:49:09.651703 30956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:49:09.653674 30941 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:49:09.653929 30941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "16f7b2e15c444168a19f174f49ae81bc"
format_stamp: "Formatted at 2026-09-26 16:49:09 on dist-test-slave-wt2j"
server_key: "20825d3f9ef46516afb251bdd30b10d3"
server_key_iv: "069414b52f33af39b8ce182fd5d27451"
server_key_version: "encryptionkey@0"
I20260926 16:49:09.654227 30941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:49:09.672613 30941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:49:09.677407 30941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:49:09.677726 30941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:49:09.700369 30941 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35877
I20260926 16:49:09.700536 31007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35877 every 8 connection(s)
I20260926 16:49:09.701843 30941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:49:09.702706 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 30941
I20260926 16:49:09.702991 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:49:09.703761 10171 external_mini_cluster.cc:1428] Setting key 0aa87715b4de4f3c85987b97f9213af9
I20260926 16:49:09.706763 31008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:49:09.719877 31008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16f7b2e15c444168a19f174f49ae81bc: Bootstrap starting.
Sep 26 16:49:09 dist-test-slave-wt2j krb5kdc[30928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 16:49:09.723380 31008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16f7b2e15c444168a19f174f49ae81bc: Neither blocks nor log segments found. Creating new log.
I20260926 16:49:09.724898 31008 log.cc:824] T 00000000000000000000000000000000 P 16f7b2e15c444168a19f174f49ae81bc: Log is configured to *not* fsync() on all Append() calls
I20260926 16:49:09.728874 31008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16f7b2e15c444168a19f174f49ae81bc: No bootstrap required, opened a new log
I20260926 16:49:09.733963 31011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:09.706780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37779 (local address 127.0.0.1:35877)
0926 16:49:09.707867 (+ 1087us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:09.707885 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:09.707944 (+ 59us) server_negotiation.cc:407] Connection header received
0926 16:49:09.710050 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:09.710082 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:09.710694 (+ 612us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:09.711266 (+ 572us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:09.713437 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:09.715106 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:09.716895 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:09.717833 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:09.718106 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:09.723680 (+ 5574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:09.723721 (+ 41us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:09.723742 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:09.723799 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:09.728332 (+ 4533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:09.729360 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:09.729373 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:09.729381 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:09.729598 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:09.730080 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:09.730088 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:09.730097 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:09.730903 (+ 806us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:09.731318 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:09.731907 (+ 589us) server_negotiation.cc:299] Negotiation successful
0926 16:49:09.732546 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":792,"thread_start_us":476,"threads_started":1}
I20260926 16:49:09.738274 31008 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16f7b2e15c444168a19f174f49ae81bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f7b2e15c444168a19f174f49ae81bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35877 } }
I20260926 16:49:09.738756 31008 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16f7b2e15c444168a19f174f49ae81bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:49:09.738894 31008 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16f7b2e15c444168a19f174f49ae81bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16f7b2e15c444168a19f174f49ae81bc, State: Initialized, Role: FOLLOWER
I20260926 16:49:09.739341 10171 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:49:09.739701 31008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16f7b2e15c444168a19f174f49ae81bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f7b2e15c444168a19f174f49ae81bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35877 } }
I20260926 16:49:09.739950 31008 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16f7b2e15c444168a19f174f49ae81bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:49:09.740103 31008 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16f7b2e15c444168a19f174f49ae81bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:49:09.740295 31008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16f7b2e15c444168a19f174f49ae81bc [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:49:09.742645 31008 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16f7b2e15c444168a19f174f49ae81bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f7b2e15c444168a19f174f49ae81bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35877 } }
I20260926 16:49:09.743182 31008 leader_election.cc:302] T 00000000000000000000000000000000 P 16f7b2e15c444168a19f174f49ae81bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16f7b2e15c444168a19f174f49ae81bc; no voters:
I20260926 16:49:09.743961 31008 leader_election.cc:288] T 00000000000000000000000000000000 P 16f7b2e15c444168a19f174f49ae81bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:49:09.744072 31016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16f7b2e15c444168a19f174f49ae81bc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:49:09.746580 31008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16f7b2e15c444168a19f174f49ae81bc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:49:09.752748 31016 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16f7b2e15c444168a19f174f49ae81bc [term 1 LEADER]: Becoming Leader. State: Replica: 16f7b2e15c444168a19f174f49ae81bc, State: Running, Role: LEADER
I20260926 16:49:09.753741 31016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16f7b2e15c444168a19f174f49ae81bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f7b2e15c444168a19f174f49ae81bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35877 } }
I20260926 16:49:09.761605 31018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16f7b2e15c444168a19f174f49ae81bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16f7b2e15c444168a19f174f49ae81bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f7b2e15c444168a19f174f49ae81bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35877 } } }
I20260926 16:49:09.762212 31018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16f7b2e15c444168a19f174f49ae81bc [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:09.765295 31011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:09.745590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45481 (local address 127.0.0.1:35877)
0926 16:49:09.745817 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:09.745822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:09.746414 (+ 592us) server_negotiation.cc:407] Connection header received
0926 16:49:09.746712 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:09.746718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:09.746820 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:09.746961 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:09.748968 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:09.750592 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:09.752436 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:09.752970 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:09.753131 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:09.756936 (+ 3805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:09.756942 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:09.756946 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:09.756990 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:09.761242 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:09.762362 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:09.762373 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:09.762377 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:09.762662 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:09.763260 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:09.763266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:09.763270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:09.763822 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:09.764163 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:09.764538 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 16:49:09.764835 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 16:49:09.766995 31019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16f7b2e15c444168a19f174f49ae81bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16f7b2e15c444168a19f174f49ae81bc. Latest consensus state: current_term: 1 leader_uuid: "16f7b2e15c444168a19f174f49ae81bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16f7b2e15c444168a19f174f49ae81bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35877 } } }
I20260926 16:49:09.767475 31019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16f7b2e15c444168a19f174f49ae81bc [sys.catalog]: This master's current role is: LEADER
W20260926 16:49:09.777231 31034 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 16f7b2e15c444168a19f174f49ae81bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:49:09.777462 31034 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:49:09.777635 31029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:49:09.779783 31029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:49:09.790339 31029 catalog_manager.cc:1347] Generated new cluster ID: 31496b48eeac49d2812861ee9bece754
I20260926 16:49:09.790508 31029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:49:09.815717 31029 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:49:09.817296 31029 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:49:09.829746 31029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16f7b2e15c444168a19f174f49ae81bc: Generated new TSK 0
I20260926 16:49:09.831111 31029 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:49:09.852182 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 30941
I20260926 16:49:09.853688 31003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:49:09.954766 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 30941
2026-09-26T16:49:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2183 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:49:10 dist-test-slave-wt2j krb5kdc[31040](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:49:10 dist-test-slave-wt2j krb5kdc[31040](info): set up 2 sockets
Sep 26 16:49:10 dist-test-slave-wt2j krb5kdc[31040](info): commencing operation
krb5kdc: starting...
W20260926 16:49:11.117388 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.002s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:49:11 dist-test-slave-wt2j krb5kdc[31040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:49:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:49:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:49:11.435500 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42611
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:49:11.853863 31053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:49:11.854277 31053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:49:11.854497 31053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:49:11.862854 31053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:49:11.863029 31053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:49:11.863142 31053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:49:11.863276 31053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:49:11.863358 31053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:49:11.875156 31053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:49:11.877180 31053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:49:11.878911 31053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:49:11.895349 31058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:11.895378 31059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:11.895438 31061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:49:11.896483 31053 server_base.cc:1034] running on GCE node
I20260926 16:49:11.897468 31053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:49:11.899179 31053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:49:11.900403 31053 hybrid_clock.cc:648] HybridClock initialized: now 1790441351900341 us; error 61 us; skew 500 ppm
I20260926 16:49:11.900832 31053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:49:11 dist-test-slave-wt2j krb5kdc[31040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:49:11.912503 31053 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:49:11.914538 31053 webserver.cc:460] Webserver started at http://127.0.0.1:34277/ using document root <none> and password file <none>
I20260926 16:49:11.915371 31053 fs_manager.cc:359] Metadata directory not provided
I20260926 16:49:11.915534 31053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:49:11.915923 31053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:49:11.919447 31053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "835ec9a31d46407dbc24561f2e91302e"
format_stamp: "Formatted at 2026-09-26 16:49:11 on dist-test-slave-wt2j"
server_key: "c3aea4e62d0ee7d85dd0607e982816f1"
server_key_iv: "53451c9e8de1d957bba96c28e81dc468"
server_key_version: "encryptionkey@0"
I20260926 16:49:11.920375 31053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "835ec9a31d46407dbc24561f2e91302e"
format_stamp: "Formatted at 2026-09-26 16:49:11 on dist-test-slave-wt2j"
server_key: "c3aea4e62d0ee7d85dd0607e982816f1"
server_key_iv: "53451c9e8de1d957bba96c28e81dc468"
server_key_version: "encryptionkey@0"
I20260926 16:49:11.926646 31053 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260926 16:49:11.932855 31068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:49:11.935328 31053 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260926 16:49:11.935639 31053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "835ec9a31d46407dbc24561f2e91302e"
format_stamp: "Formatted at 2026-09-26 16:49:11 on dist-test-slave-wt2j"
server_key: "c3aea4e62d0ee7d85dd0607e982816f1"
server_key_iv: "53451c9e8de1d957bba96c28e81dc468"
server_key_version: "encryptionkey@0"
I20260926 16:49:11.935958 31053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:49:11.963361 31053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:49:11.968503 31053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:49:11.968775 31053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:49:11.989773 31053 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42611
I20260926 16:49:11.989837 31119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42611 every 8 connection(s)
I20260926 16:49:11.991729 31053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:49:11.996665 31120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:49:11.998860 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 31053
I20260926 16:49:11.999089 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:49:11.999768 10171 external_mini_cluster.cc:1428] Setting key e9848ecc0724cdf277fa4a54b2023cdb
I20260926 16:49:12.014979 31120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 835ec9a31d46407dbc24561f2e91302e: Bootstrap starting.
I20260926 16:49:12.018575 31123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:12.003376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33119 (local address 127.0.0.1:42611)
0926 16:49:12.005866 (+ 2490us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:12.005888 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:12.005952 (+ 64us) server_negotiation.cc:407] Connection header received
0926 16:49:12.008546 (+ 2594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:12.008586 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:12.009264 (+ 678us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:12.009712 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:12.010340 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:12.012133 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:12.013547 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:12.014292 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:12.014518 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:12.015998 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:12.016006 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:12.016025 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:12.016653 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:12.016748 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:12.016776 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:12.016923 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:12.017152 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 16:49:12.017460 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2117,"thread_start_us":1624,"threads_started":1}
I20260926 16:49:12.019953 31120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 835ec9a31d46407dbc24561f2e91302e: Neither blocks nor log segments found. Creating new log.
I20260926 16:49:12.021574 31120 log.cc:824] T 00000000000000000000000000000000 P 835ec9a31d46407dbc24561f2e91302e: Log is configured to *not* fsync() on all Append() calls
I20260926 16:49:12.023630 10171 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:49:12.025631 31120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 835ec9a31d46407dbc24561f2e91302e: No bootstrap required, opened a new log
I20260926 16:49:12.036877 31123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:12.028575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48951 (local address 127.0.0.1:42611)
0926 16:49:12.028890 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:12.028897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:12.029416 (+ 519us) server_negotiation.cc:407] Connection header received
0926 16:49:12.029669 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:12.029674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:12.029767 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:12.029874 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:12.030641 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:12.031514 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:12.033785 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:12.034419 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:12.034561 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:12.036093 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:12.036098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:12.036102 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:12.036123 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:12.036167 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:12.036185 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:12.036266 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:12.036447 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 16:49:12.036530 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 16:49:12.037338 31120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 835ec9a31d46407dbc24561f2e91302e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "835ec9a31d46407dbc24561f2e91302e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42611 } }
I20260926 16:49:12.037987 31120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 835ec9a31d46407dbc24561f2e91302e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:49:12.038112 31120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 835ec9a31d46407dbc24561f2e91302e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835ec9a31d46407dbc24561f2e91302e, State: Initialized, Role: FOLLOWER
I20260926 16:49:12.038936 31120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 835ec9a31d46407dbc24561f2e91302e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "835ec9a31d46407dbc24561f2e91302e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42611 } }
I20260926 16:49:12.039201 31120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 835ec9a31d46407dbc24561f2e91302e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:49:12.039330 31120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 835ec9a31d46407dbc24561f2e91302e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:49:12.039479 31120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 835ec9a31d46407dbc24561f2e91302e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:49:12.042765 31120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 835ec9a31d46407dbc24561f2e91302e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "835ec9a31d46407dbc24561f2e91302e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42611 } }
I20260926 16:49:12.043365 31120 leader_election.cc:302] T 00000000000000000000000000000000 P 835ec9a31d46407dbc24561f2e91302e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 835ec9a31d46407dbc24561f2e91302e; no voters:
I20260926 16:49:12.044097 31120 leader_election.cc:288] T 00000000000000000000000000000000 P 835ec9a31d46407dbc24561f2e91302e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:49:12.044364 31130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 835ec9a31d46407dbc24561f2e91302e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:49:12.045275 31130 raft_consensus.cc:686] T 00000000000000000000000000000000 P 835ec9a31d46407dbc24561f2e91302e [term 1 LEADER]: Becoming Leader. State: Replica: 835ec9a31d46407dbc24561f2e91302e, State: Running, Role: LEADER
I20260926 16:49:12.045931 31130 consensus_queue.cc:237] T 00000000000000000000000000000000 P 835ec9a31d46407dbc24561f2e91302e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "835ec9a31d46407dbc24561f2e91302e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42611 } }
I20260926 16:49:12.046909 31120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 835ec9a31d46407dbc24561f2e91302e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:49:12.048863 31132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 835ec9a31d46407dbc24561f2e91302e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 835ec9a31d46407dbc24561f2e91302e. Latest consensus state: current_term: 1 leader_uuid: "835ec9a31d46407dbc24561f2e91302e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "835ec9a31d46407dbc24561f2e91302e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42611 } } }
I20260926 16:49:12.049618 31131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 835ec9a31d46407dbc24561f2e91302e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "835ec9a31d46407dbc24561f2e91302e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "835ec9a31d46407dbc24561f2e91302e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42611 } } }
I20260926 16:49:12.049938 31132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 835ec9a31d46407dbc24561f2e91302e [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:12.050248 31131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 835ec9a31d46407dbc24561f2e91302e [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:12.051429 31136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:49:12.060286 31136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:49:12.071815 31136 catalog_manager.cc:1347] Generated new cluster ID: 63b4f4db1eed49e3ac5456f9110c1900
I20260926 16:49:12.071960 31136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:49:12.096530 31136 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:49:12.098433 31136 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:49:12.107756 31136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 835ec9a31d46407dbc24561f2e91302e: Generated new TSK 0
I20260926 16:49:12.108927 31136 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:49:12.116559 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 31053
W20260926 16:49:12.118005 31115 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33119
W20260926 16:49:12.118904 10171 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 31053: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:49:12.119077 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 31053
2026-09-26T16:49:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2162 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:49:12 dist-test-slave-wt2j krb5kdc[31151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:49:12 dist-test-slave-wt2j krb5kdc[31151](info): set up 2 sockets
Sep 26 16:49:12 dist-test-slave-wt2j krb5kdc[31151](info): commencing operation
krb5kdc: starting...
W20260926 16:49:13.273406 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.113s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:49:13 dist-test-slave-wt2j krb5kdc[31151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:49:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:49:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:49:13.591979 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35051
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:49:13.958313 31164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:49:13.958618 31164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:49:13.958711 31164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:49:13.966248 31164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:49:13.966374 31164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:49:13.966421 31164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:49:13.966475 31164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:49:13.966524 31164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:49:13.977270 31164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:49:13.978598 31164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:49:13.979945 31164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:49:13.993477 31169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:13.993626 31170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:13.993645 31172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:49:13.995637 31164 server_base.cc:1034] running on GCE node
I20260926 16:49:13.996558 31164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:49:13.998214 31164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:49:14.001662 31164 hybrid_clock.cc:648] HybridClock initialized: now 1790441354001573 us; error 53 us; skew 500 ppm
I20260926 16:49:14.002269 31164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:49:14 dist-test-slave-wt2j krb5kdc[31151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:49:14.007236 31164 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:49:14.009420 31164 webserver.cc:460] Webserver started at http://127.0.0.1:36265/ using document root <none> and password file <none>
I20260926 16:49:14.010197 31164 fs_manager.cc:359] Metadata directory not provided
I20260926 16:49:14.010313 31164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:49:14.010656 31164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:49:14.013890 31164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "98ae61206f614e10ae51cc89939fc0d3"
format_stamp: "Formatted at 2026-09-26 16:49:14 on dist-test-slave-wt2j"
server_key: "c00ff1b7ce51a21cce9ad926b41f09b6"
server_key_iv: "fbe1fb5e5b4d8d4c480585944373f413"
server_key_version: "encryptionkey@0"
I20260926 16:49:14.014720 31164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "98ae61206f614e10ae51cc89939fc0d3"
format_stamp: "Formatted at 2026-09-26 16:49:14 on dist-test-slave-wt2j"
server_key: "c00ff1b7ce51a21cce9ad926b41f09b6"
server_key_iv: "fbe1fb5e5b4d8d4c480585944373f413"
server_key_version: "encryptionkey@0"
I20260926 16:49:14.020766 31164 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260926 16:49:14.025059 31179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:49:14.026873 31164 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 16:49:14.027124 31164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "98ae61206f614e10ae51cc89939fc0d3"
format_stamp: "Formatted at 2026-09-26 16:49:14 on dist-test-slave-wt2j"
server_key: "c00ff1b7ce51a21cce9ad926b41f09b6"
server_key_iv: "fbe1fb5e5b4d8d4c480585944373f413"
server_key_version: "encryptionkey@0"
I20260926 16:49:14.027356 31164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:49:14.033704 31164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:49:14.038466 31164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:49:14.038789 31164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:49:14.060837 31164 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35051
I20260926 16:49:14.060932 31230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35051 every 8 connection(s)
I20260926 16:49:14.062348 31164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:49:14.063432 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 31164
I20260926 16:49:14.063685 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:49:14.064337 10171 external_mini_cluster.cc:1428] Setting key ea25db9de47b8836e4b0f30c9e35239c
I20260926 16:49:14.068308 31231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:49:14.076434 31234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:14.066592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42303 (local address 127.0.0.1:35051)
0926 16:49:14.067917 (+ 1325us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:14.067937 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:14.068313 (+ 376us) server_negotiation.cc:407] Connection header received
0926 16:49:14.071150 (+ 2837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:14.071183 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:14.071729 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:14.072314 (+ 585us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:14.073664 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:14.073673 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:14.073693 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:14.073828 (+ 135us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:14.073972 (+ 144us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:14.074022 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:14.074405 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:14.074568 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 16:49:14.074841 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":963,"thread_start_us":595,"threads_started":1}
I20260926 16:49:14.080994 10171 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:49:14.084290 31231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ae61206f614e10ae51cc89939fc0d3: Bootstrap starting.
I20260926 16:49:14.088881 31234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:14.085635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32869 (local address 127.0.0.1:35051)
0926 16:49:14.085867 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:14.085873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:14.086317 (+ 444us) server_negotiation.cc:407] Connection header received
0926 16:49:14.086582 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:14.086586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:14.086651 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:14.086759 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:14.088041 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:14.088045 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:14.088048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:14.088065 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:14.088111 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:14.088128 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:14.088232 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:14.088440 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 16:49:14.088514 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 16:49:14.089192 31231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98ae61206f614e10ae51cc89939fc0d3: Neither blocks nor log segments found. Creating new log.
I20260926 16:49:14.090595 31231 log.cc:824] T 00000000000000000000000000000000 P 98ae61206f614e10ae51cc89939fc0d3: Log is configured to *not* fsync() on all Append() calls
I20260926 16:49:14.094370 31231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ae61206f614e10ae51cc89939fc0d3: No bootstrap required, opened a new log
I20260926 16:49:14.103641 31231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98ae61206f614e10ae51cc89939fc0d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ae61206f614e10ae51cc89939fc0d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35051 } }
I20260926 16:49:14.104244 31231 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98ae61206f614e10ae51cc89939fc0d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:49:14.104374 31231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98ae61206f614e10ae51cc89939fc0d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98ae61206f614e10ae51cc89939fc0d3, State: Initialized, Role: FOLLOWER
I20260926 16:49:14.105178 31231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98ae61206f614e10ae51cc89939fc0d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ae61206f614e10ae51cc89939fc0d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35051 } }
I20260926 16:49:14.105484 31231 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98ae61206f614e10ae51cc89939fc0d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:49:14.105664 31231 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98ae61206f614e10ae51cc89939fc0d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:49:14.105885 31231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98ae61206f614e10ae51cc89939fc0d3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:49:14.108861 31231 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98ae61206f614e10ae51cc89939fc0d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ae61206f614e10ae51cc89939fc0d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35051 } }
I20260926 16:49:14.109635 31231 leader_election.cc:302] T 00000000000000000000000000000000 P 98ae61206f614e10ae51cc89939fc0d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98ae61206f614e10ae51cc89939fc0d3; no voters:
I20260926 16:49:14.110411 31231 leader_election.cc:288] T 00000000000000000000000000000000 P 98ae61206f614e10ae51cc89939fc0d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:49:14.110641 31241 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98ae61206f614e10ae51cc89939fc0d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:49:14.111577 31241 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98ae61206f614e10ae51cc89939fc0d3 [term 1 LEADER]: Becoming Leader. State: Replica: 98ae61206f614e10ae51cc89939fc0d3, State: Running, Role: LEADER
I20260926 16:49:14.112639 31241 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98ae61206f614e10ae51cc89939fc0d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ae61206f614e10ae51cc89939fc0d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35051 } }
I20260926 16:49:14.113384 31231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98ae61206f614e10ae51cc89939fc0d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:49:14.116986 31242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ae61206f614e10ae51cc89939fc0d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98ae61206f614e10ae51cc89939fc0d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ae61206f614e10ae51cc89939fc0d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35051 } } }
I20260926 16:49:14.117659 31242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ae61206f614e10ae51cc89939fc0d3 [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:14.118338 31243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ae61206f614e10ae51cc89939fc0d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98ae61206f614e10ae51cc89939fc0d3. Latest consensus state: current_term: 1 leader_uuid: "98ae61206f614e10ae51cc89939fc0d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ae61206f614e10ae51cc89939fc0d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35051 } } }
I20260926 16:49:14.118736 31243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ae61206f614e10ae51cc89939fc0d3 [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:14.120206 31250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:49:14.126811 31250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:49:14.138958 31250 catalog_manager.cc:1347] Generated new cluster ID: d214b763713b4929b0a50e0b4eb07c00
I20260926 16:49:14.139209 31250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:49:14.164568 31250 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:49:14.165199 31250 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:49:14.181566 31250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98ae61206f614e10ae51cc89939fc0d3: Generated new TSK 0
I20260926 16:49:14.182942 31250 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:49:14.207378 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 31164
W20260926 16:49:14.208961 31226 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42303
W20260926 16:49:14.209898 10171 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 31164: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:49:14.210083 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 31164
2026-09-26T16:49:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2091 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:49:14 dist-test-slave-wt2j krb5kdc[31262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:49:14 dist-test-slave-wt2j krb5kdc[31262](info): set up 2 sockets
Sep 26 16:49:14 dist-test-slave-wt2j krb5kdc[31262](info): commencing operation
krb5kdc: starting...
W20260926 16:49:15.377823 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.007s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:49:15 dist-test-slave-wt2j krb5kdc[31262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:49:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:49:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:49:15.704703 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42623
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:49:16.151714 31275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:49:16.152112 31275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:49:16.152216 31275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:49:16.160800 31275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:49:16.161003 31275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:49:16.161080 31275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:49:16.161160 31275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:49:16.161211 31275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:49:16.172801 31275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:49:16.174499 31275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:49:16.176223 31275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:49:16.190863 31280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:16.190979 31281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:16.190928 31283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:49:16.192574 31275 server_base.cc:1034] running on GCE node
I20260926 16:49:16.193578 31275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:49:16.195374 31275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:49:16.196609 31275 hybrid_clock.cc:648] HybridClock initialized: now 1790441356196551 us; error 48 us; skew 500 ppm
I20260926 16:49:16.197005 31275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:49:16 dist-test-slave-wt2j krb5kdc[31262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:49:16.200925 31275 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:49:16.202845 31275 webserver.cc:460] Webserver started at http://127.0.0.1:39143/ using document root <none> and password file <none>
I20260926 16:49:16.203797 31275 fs_manager.cc:359] Metadata directory not provided
I20260926 16:49:16.203959 31275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:49:16.204329 31275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:49:16.207949 31275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "bef5757df07046c0b447ab0c0d8acbca"
format_stamp: "Formatted at 2026-09-26 16:49:16 on dist-test-slave-wt2j"
server_key: "3410f28dcb6c531673d7afa51b19d6e6"
server_key_iv: "f33dbe594e5611762cd529b301ced7b6"
server_key_version: "encryptionkey@0"
I20260926 16:49:16.209000 31275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "bef5757df07046c0b447ab0c0d8acbca"
format_stamp: "Formatted at 2026-09-26 16:49:16 on dist-test-slave-wt2j"
server_key: "3410f28dcb6c531673d7afa51b19d6e6"
server_key_iv: "f33dbe594e5611762cd529b301ced7b6"
server_key_version: "encryptionkey@0"
I20260926 16:49:16.215523 31275 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 16:49:16.220091 31290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:49:16.221999 31275 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260926 16:49:16.222201 31275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "bef5757df07046c0b447ab0c0d8acbca"
format_stamp: "Formatted at 2026-09-26 16:49:16 on dist-test-slave-wt2j"
server_key: "3410f28dcb6c531673d7afa51b19d6e6"
server_key_iv: "f33dbe594e5611762cd529b301ced7b6"
server_key_version: "encryptionkey@0"
I20260926 16:49:16.222414 31275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:49:16.235145 31275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:49:16.240414 31275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:49:16.240705 31275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:49:16.263823 31275 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42623
I20260926 16:49:16.263878 31341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42623 every 8 connection(s)
I20260926 16:49:16.265619 31275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:49:16.268800 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 31275
I20260926 16:49:16.269011 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:49:16.269527 10171 external_mini_cluster.cc:1428] Setting key 1e3ad8a7e146793c59fd858f3133fccc
I20260926 16:49:16.270342 31342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:49:16.286307 31342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bef5757df07046c0b447ab0c0d8acbca: Bootstrap starting.
Sep 26 16:49:16 dist-test-slave-wt2j krb5kdc[31262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 16:49:16.290381 31342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bef5757df07046c0b447ab0c0d8acbca: Neither blocks nor log segments found. Creating new log.
I20260926 16:49:16.291718 31342 log.cc:824] T 00000000000000000000000000000000 P bef5757df07046c0b447ab0c0d8acbca: Log is configured to *not* fsync() on all Append() calls
I20260926 16:49:16.295799 31342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bef5757df07046c0b447ab0c0d8acbca: No bootstrap required, opened a new log
I20260926 16:49:16.299690 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:16.271996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:59843 (local address 127.0.0.1:42623)
0926 16:49:16.272944 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:16.272962 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:16.273033 (+ 71us) server_negotiation.cc:407] Connection header received
0926 16:49:16.275851 (+ 2818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:16.275883 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:16.276486 (+ 603us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:16.276996 (+ 510us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:16.278514 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:16.280764 (+ 2250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:16.282255 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:16.283480 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:16.283740 (+ 260us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:16.289186 (+ 5446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:16.289198 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:16.289224 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:16.289280 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:16.293600 (+ 4320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:16.294847 (+ 1247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:16.294855 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:16.294863 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:16.295074 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:16.295744 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:16.295750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:16.295753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:16.296674 (+ 921us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:16.296990 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:16.297672 (+ 682us) server_negotiation.cc:299] Negotiation successful
0926 16:49:16.298176 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":376,"threads_started":1}
I20260926 16:49:16.304584 10171 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:49:16.305115 31342 raft_consensus.cc:348] T 00000000000000000000000000000000 P bef5757df07046c0b447ab0c0d8acbca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef5757df07046c0b447ab0c0d8acbca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42623 } }
I20260926 16:49:16.305531 31342 raft_consensus.cc:374] T 00000000000000000000000000000000 P bef5757df07046c0b447ab0c0d8acbca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:49:16.305637 31342 raft_consensus.cc:729] T 00000000000000000000000000000000 P bef5757df07046c0b447ab0c0d8acbca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bef5757df07046c0b447ab0c0d8acbca, State: Initialized, Role: FOLLOWER
I20260926 16:49:16.306293 31342 consensus_queue.cc:260] T 00000000000000000000000000000000 P bef5757df07046c0b447ab0c0d8acbca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef5757df07046c0b447ab0c0d8acbca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42623 } }
I20260926 16:49:16.306528 31342 raft_consensus.cc:388] T 00000000000000000000000000000000 P bef5757df07046c0b447ab0c0d8acbca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:49:16.306689 31342 raft_consensus.cc:482] T 00000000000000000000000000000000 P bef5757df07046c0b447ab0c0d8acbca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:49:16.306859 31342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bef5757df07046c0b447ab0c0d8acbca [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:49:16.309396 31342 raft_consensus.cc:504] T 00000000000000000000000000000000 P bef5757df07046c0b447ab0c0d8acbca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef5757df07046c0b447ab0c0d8acbca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42623 } }
I20260926 16:49:16.309945 31342 leader_election.cc:302] T 00000000000000000000000000000000 P bef5757df07046c0b447ab0c0d8acbca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bef5757df07046c0b447ab0c0d8acbca; no voters:
I20260926 16:49:16.310650 31342 leader_election.cc:288] T 00000000000000000000000000000000 P bef5757df07046c0b447ab0c0d8acbca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:49:16.310820 31351 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bef5757df07046c0b447ab0c0d8acbca [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:49:16.314141 31342 sys_catalog.cc:564] T 00000000000000000000000000000000 P bef5757df07046c0b447ab0c0d8acbca [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:49:16.314810 31351 raft_consensus.cc:686] T 00000000000000000000000000000000 P bef5757df07046c0b447ab0c0d8acbca [term 1 LEADER]: Becoming Leader. State: Replica: bef5757df07046c0b447ab0c0d8acbca, State: Running, Role: LEADER
I20260926 16:49:16.315408 31351 consensus_queue.cc:237] T 00000000000000000000000000000000 P bef5757df07046c0b447ab0c0d8acbca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef5757df07046c0b447ab0c0d8acbca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42623 } }
I20260926 16:49:16.320421 31353 sys_catalog.cc:455] T 00000000000000000000000000000000 P bef5757df07046c0b447ab0c0d8acbca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bef5757df07046c0b447ab0c0d8acbca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef5757df07046c0b447ab0c0d8acbca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42623 } } }
I20260926 16:49:16.321045 31353 sys_catalog.cc:458] T 00000000000000000000000000000000 P bef5757df07046c0b447ab0c0d8acbca [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:16.321763 31354 sys_catalog.cc:455] T 00000000000000000000000000000000 P bef5757df07046c0b447ab0c0d8acbca [sys.catalog]: SysCatalogTable state changed. Reason: New leader bef5757df07046c0b447ab0c0d8acbca. Latest consensus state: current_term: 1 leader_uuid: "bef5757df07046c0b447ab0c0d8acbca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bef5757df07046c0b447ab0c0d8acbca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42623 } } }
I20260926 16:49:16.322201 31354 sys_catalog.cc:458] T 00000000000000000000000000000000 P bef5757df07046c0b447ab0c0d8acbca [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:16.327651 31361 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:49:16.335436 31361 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:49:16.342325 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:16.310943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:36389 (local address 127.0.0.1:42623)
0926 16:49:16.311279 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:16.311291 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:16.318466 (+ 7175us) server_negotiation.cc:407] Connection header received
0926 16:49:16.318772 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:16.318779 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:16.318895 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:16.319078 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:16.322221 (+ 3143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:16.324179 (+ 1958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:16.327112 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:16.328054 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:16.328217 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:16.331508 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:16.331515 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:16.331522 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:16.331601 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:16.336987 (+ 5386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:16.338815 (+ 1828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:16.338824 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:16.338830 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:16.339061 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:16.339901 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:16.339912 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:16.339918 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:16.340491 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:16.340789 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:16.341606 (+ 817us) server_negotiation.cc:299] Negotiation successful
0926 16:49:16.341875 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260926 16:49:16.351184 31361 catalog_manager.cc:1347] Generated new cluster ID: 445dc5f00f7a4c84ae7ab85697d15aad
I20260926 16:49:16.351425 31361 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:49:16.374209 31361 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:49:16.376214 31361 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:49:16.386368 31361 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bef5757df07046c0b447ab0c0d8acbca: Generated new TSK 0
I20260926 16:49:16.387271 31361 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:49:16.398392 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 31275
I20260926 16:49:16.400189 31337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:49:16.499425 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 31275
W20260926 16:49:16.518723 31260 connection.cc:535] client connection to 127.0.0.1:42623 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42623): Connection reset by peer (error 104)
2026-09-26T16:49:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:49:16 dist-test-slave-wt2j krb5kdc[31374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:49:16 dist-test-slave-wt2j krb5kdc[31374](info): set up 2 sockets
Sep 26 16:49:16 dist-test-slave-wt2j krb5kdc[31374](info): commencing operation
krb5kdc: starting...
W20260926 16:49:17.698307 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.001s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:49:17 dist-test-slave-wt2j krb5kdc[31374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:49:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:49:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:49:18.028234 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38631
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:49:18.453606 31387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:49:18.454005 31387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:49:18.454172 31387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:49:18.463518 31387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:49:18.463744 31387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:49:18.463860 31387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:49:18.463979 31387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:49:18.464066 31387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:49:18.475149 31387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:49:18.476739 31387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:49:18.478176 31387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:49:18.493103 31393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:18.493070 31392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:18.493424 31395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:49:18.493780 31387 server_base.cc:1034] running on GCE node
I20260926 16:49:18.494889 31387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:49:18.496781 31387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:49:18.498114 31387 hybrid_clock.cc:648] HybridClock initialized: now 1790441358498030 us; error 62 us; skew 500 ppm
I20260926 16:49:18.498610 31387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:49:18 dist-test-slave-wt2j krb5kdc[31374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:49:18.502393 31387 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:49:18.504319 31387 webserver.cc:460] Webserver started at http://127.0.0.1:42725/ using document root <none> and password file <none>
I20260926 16:49:18.505124 31387 fs_manager.cc:359] Metadata directory not provided
I20260926 16:49:18.505286 31387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:49:18.505631 31387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:49:18.508997 31387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "4bee66d6bdc54d02b8d3bc18a21c2a8a"
format_stamp: "Formatted at 2026-09-26 16:49:18 on dist-test-slave-wt2j"
server_key: "a44e604331f43dd9420e7a31c99429d2"
server_key_iv: "67c0ff608619c8398c6a43e10d384f43"
server_key_version: "encryptionkey@0"
I20260926 16:49:18.509865 31387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "4bee66d6bdc54d02b8d3bc18a21c2a8a"
format_stamp: "Formatted at 2026-09-26 16:49:18 on dist-test-slave-wt2j"
server_key: "a44e604331f43dd9420e7a31c99429d2"
server_key_iv: "67c0ff608619c8398c6a43e10d384f43"
server_key_version: "encryptionkey@0"
I20260926 16:49:18.516294 31387 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 16:49:18.520799 31402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:49:18.522811 31387 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260926 16:49:18.523077 31387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "4bee66d6bdc54d02b8d3bc18a21c2a8a"
format_stamp: "Formatted at 2026-09-26 16:49:18 on dist-test-slave-wt2j"
server_key: "a44e604331f43dd9420e7a31c99429d2"
server_key_iv: "67c0ff608619c8398c6a43e10d384f43"
server_key_version: "encryptionkey@0"
I20260926 16:49:18.523316 31387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:49:18.560374 31387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:49:18.566032 31387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:49:18.566310 31387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:49:18.588299 31387 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38631
I20260926 16:49:18.588465 31453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38631 every 8 connection(s)
I20260926 16:49:18.589994 31387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:49:18.595309 31454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:49:18.595999 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 31387
I20260926 16:49:18.596220 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:49:18.596903 10171 external_mini_cluster.cc:1428] Setting key 8e644a691bde17f36824501be3be03f8
I20260926 16:49:18.611366 31454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bee66d6bdc54d02b8d3bc18a21c2a8a: Bootstrap starting.
I20260926 16:49:18.612931 31457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:18.599352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:60193 (local address 127.0.0.1:38631)
0926 16:49:18.600513 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:18.600529 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:18.600579 (+ 50us) server_negotiation.cc:407] Connection header received
0926 16:49:18.603398 (+ 2819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:18.603441 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:18.604013 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:18.604528 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:18.605148 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:18.606538 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:18.607995 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:18.608802 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:18.609047 (+ 245us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:18.610415 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:18.610423 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:18.610441 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:18.611063 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:18.611164 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:18.611190 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:18.611373 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:18.611680 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 16:49:18.611880 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":863,"thread_start_us":464,"threads_started":1}
I20260926 16:49:18.616053 31454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bee66d6bdc54d02b8d3bc18a21c2a8a: Neither blocks nor log segments found. Creating new log.
I20260926 16:49:18.617487 31454 log.cc:824] T 00000000000000000000000000000000 P 4bee66d6bdc54d02b8d3bc18a21c2a8a: Log is configured to *not* fsync() on all Append() calls
I20260926 16:49:18.618122 10171 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:49:18.620999 31454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bee66d6bdc54d02b8d3bc18a21c2a8a: No bootstrap required, opened a new log
I20260926 16:49:18.630007 31457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:18.622939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:38251 (local address 127.0.0.1:38631)
0926 16:49:18.623242 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:18.623251 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:18.623868 (+ 617us) server_negotiation.cc:407] Connection header received
0926 16:49:18.624127 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:18.624131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:18.624209 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:18.624329 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:18.624886 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:18.625720 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:18.627116 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:18.627824 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:18.627971 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:18.629197 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:18.629203 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:18.629206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:18.629228 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:18.629285 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:18.629300 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:18.629440 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:18.629628 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 16:49:18.629691 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260926 16:49:18.631641 31454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bee66d6bdc54d02b8d3bc18a21c2a8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bee66d6bdc54d02b8d3bc18a21c2a8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38631 } }
I20260926 16:49:18.632265 31454 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bee66d6bdc54d02b8d3bc18a21c2a8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:49:18.632406 31454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bee66d6bdc54d02b8d3bc18a21c2a8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bee66d6bdc54d02b8d3bc18a21c2a8a, State: Initialized, Role: FOLLOWER
I20260926 16:49:18.633196 31454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bee66d6bdc54d02b8d3bc18a21c2a8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bee66d6bdc54d02b8d3bc18a21c2a8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38631 } }
I20260926 16:49:18.633489 31454 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bee66d6bdc54d02b8d3bc18a21c2a8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:49:18.633666 31454 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bee66d6bdc54d02b8d3bc18a21c2a8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:49:18.633873 31454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bee66d6bdc54d02b8d3bc18a21c2a8a [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:49:18.637364 31454 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bee66d6bdc54d02b8d3bc18a21c2a8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bee66d6bdc54d02b8d3bc18a21c2a8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38631 } }
I20260926 16:49:18.638136 31454 leader_election.cc:302] T 00000000000000000000000000000000 P 4bee66d6bdc54d02b8d3bc18a21c2a8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bee66d6bdc54d02b8d3bc18a21c2a8a; no voters:
I20260926 16:49:18.639062 31454 leader_election.cc:288] T 00000000000000000000000000000000 P 4bee66d6bdc54d02b8d3bc18a21c2a8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:49:18.639250 31464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bee66d6bdc54d02b8d3bc18a21c2a8a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:49:18.640203 31464 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bee66d6bdc54d02b8d3bc18a21c2a8a [term 1 LEADER]: Becoming Leader. State: Replica: 4bee66d6bdc54d02b8d3bc18a21c2a8a, State: Running, Role: LEADER
I20260926 16:49:18.640841 31464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bee66d6bdc54d02b8d3bc18a21c2a8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bee66d6bdc54d02b8d3bc18a21c2a8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38631 } }
I20260926 16:49:18.641960 31454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bee66d6bdc54d02b8d3bc18a21c2a8a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:49:18.644699 31465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bee66d6bdc54d02b8d3bc18a21c2a8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bee66d6bdc54d02b8d3bc18a21c2a8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bee66d6bdc54d02b8d3bc18a21c2a8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38631 } } }
I20260926 16:49:18.644699 31466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bee66d6bdc54d02b8d3bc18a21c2a8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bee66d6bdc54d02b8d3bc18a21c2a8a. Latest consensus state: current_term: 1 leader_uuid: "4bee66d6bdc54d02b8d3bc18a21c2a8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bee66d6bdc54d02b8d3bc18a21c2a8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38631 } } }
I20260926 16:49:18.645283 31465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bee66d6bdc54d02b8d3bc18a21c2a8a [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:18.645356 31466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bee66d6bdc54d02b8d3bc18a21c2a8a [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:18.646239 31470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:49:18.653012 31470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:49:18.664734 31470 catalog_manager.cc:1347] Generated new cluster ID: 9fdf704567a8462599d866e05800f119
I20260926 16:49:18.664878 31470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:49:18.702527 31470 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:49:18.704521 31470 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:49:18.732277 31470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bee66d6bdc54d02b8d3bc18a21c2a8a: Generated new TSK 0
I20260926 16:49:18.733605 31470 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:49:18.752889 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 31387
W20260926 16:49:18.755316 31449 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.144:60193
W20260926 16:49:18.756429 10171 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 31387: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:49:18.756620 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 31387
W20260926 16:49:18.775399 31372 connection.cc:535] client connection to 127.0.0.1:38631 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38631): Connection reset by peer (error 104)
2026-09-26T16:49:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2256 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:49:18 dist-test-slave-wt2j krb5kdc[31485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:49:18 dist-test-slave-wt2j krb5kdc[31485](info): set up 2 sockets
Sep 26 16:49:18 dist-test-slave-wt2j krb5kdc[31485](info): commencing operation
krb5kdc: starting...
W20260926 16:49:19.930034 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:49:20 dist-test-slave-wt2j krb5kdc[31485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:49:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:49:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:49:20.256451 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40283
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:49:20.618846 31498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:49:20.619184 31498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:49:20.619314 31498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:49:20.626731 31498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:49:20.626871 31498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:49:20.626924 31498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:49:20.627223 31498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:49:20.627318 31498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:49:20.638254 31498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:49:20.639688 31498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:49:20.641029 31498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:49:20.655140 31504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:20.655076 31506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:20.655154 31503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:49:20.656163 31498 server_base.cc:1034] running on GCE node
I20260926 16:49:20.657253 31498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:49:20.659446 31498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:49:20.660764 31498 hybrid_clock.cc:648] HybridClock initialized: now 1790441360660696 us; error 60 us; skew 500 ppm
I20260926 16:49:20.661276 31498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:49:20 dist-test-slave-wt2j krb5kdc[31485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:49:20.665433 31498 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:49:20.667426 31498 webserver.cc:460] Webserver started at http://127.0.0.1:45679/ using document root <none> and password file <none>
I20260926 16:49:20.668444 31498 fs_manager.cc:359] Metadata directory not provided
I20260926 16:49:20.668604 31498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:49:20.668990 31498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:49:20.672780 31498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "418cf4276bc44e41803e2ae394fdf09f"
format_stamp: "Formatted at 2026-09-26 16:49:20 on dist-test-slave-wt2j"
server_key: "55e07dee2c6790d3a99c779187eaa6d8"
server_key_iv: "d27a5683f9bd085720b3096d3ff1ed08"
server_key_version: "encryptionkey@0"
I20260926 16:49:20.673815 31498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "418cf4276bc44e41803e2ae394fdf09f"
format_stamp: "Formatted at 2026-09-26 16:49:20 on dist-test-slave-wt2j"
server_key: "55e07dee2c6790d3a99c779187eaa6d8"
server_key_iv: "d27a5683f9bd085720b3096d3ff1ed08"
server_key_version: "encryptionkey@0"
I20260926 16:49:20.680863 31498 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260926 16:49:20.685849 31513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:49:20.688072 31498 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260926 16:49:20.688488 31498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "418cf4276bc44e41803e2ae394fdf09f"
format_stamp: "Formatted at 2026-09-26 16:49:20 on dist-test-slave-wt2j"
server_key: "55e07dee2c6790d3a99c779187eaa6d8"
server_key_iv: "d27a5683f9bd085720b3096d3ff1ed08"
server_key_version: "encryptionkey@0"
I20260926 16:49:20.688807 31498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:49:20.698612 31498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:49:20.704222 31498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:49:20.704644 31498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:49:20.728462 31498 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40283
I20260926 16:49:20.728521 31564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40283 every 8 connection(s)
I20260926 16:49:20.730259 31498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:49:20.734688 31565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:49:20.739832 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 31498
I20260926 16:49:20.740037 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:49:20.740650 10171 external_mini_cluster.cc:1428] Setting key 7fca57c4064dbaf983b65dbbadc08cf2
I20260926 16:49:20.746907 31565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 418cf4276bc44e41803e2ae394fdf09f: Bootstrap starting.
I20260926 16:49:20.750478 31568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:20.743120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:45867 (local address 127.0.0.1:40283)
0926 16:49:20.744218 (+ 1098us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:20.744233 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:20.744295 (+ 62us) server_negotiation.cc:407] Connection header received
0926 16:49:20.746382 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:20.746412 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:20.746879 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:20.747359 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:20.748553 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:20.748560 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:20.748574 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:20.748591 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:20.748659 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:20.748680 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:20.748900 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:20.749023 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 16:49:20.749311 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":517,"threads_started":1}
I20260926 16:49:20.750921 31565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 418cf4276bc44e41803e2ae394fdf09f: Neither blocks nor log segments found. Creating new log.
I20260926 16:49:20.752159 31565 log.cc:824] T 00000000000000000000000000000000 P 418cf4276bc44e41803e2ae394fdf09f: Log is configured to *not* fsync() on all Append() calls
I20260926 16:49:20.755721 10171 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:49:20.755782 31565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 418cf4276bc44e41803e2ae394fdf09f: No bootstrap required, opened a new log
I20260926 16:49:20.764534 31565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 418cf4276bc44e41803e2ae394fdf09f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "418cf4276bc44e41803e2ae394fdf09f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40283 } }
I20260926 16:49:20.765198 31565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 418cf4276bc44e41803e2ae394fdf09f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:49:20.765306 31565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 418cf4276bc44e41803e2ae394fdf09f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 418cf4276bc44e41803e2ae394fdf09f, State: Initialized, Role: FOLLOWER
I20260926 16:49:20.765309 31568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:20.761846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:50417 (local address 127.0.0.1:40283)
0926 16:49:20.762075 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:20.762084 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:20.762536 (+ 452us) server_negotiation.cc:407] Connection header received
0926 16:49:20.762810 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:20.762835 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:20.762921 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:20.763044 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:20.764331 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:20.764341 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:20.764347 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:20.764376 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:20.764509 (+ 133us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:20.764537 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:20.764684 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:20.764825 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 16:49:20.764900 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:49:20.766012 31565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 418cf4276bc44e41803e2ae394fdf09f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "418cf4276bc44e41803e2ae394fdf09f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40283 } }
I20260926 16:49:20.766353 31565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 418cf4276bc44e41803e2ae394fdf09f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:49:20.766521 31565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 418cf4276bc44e41803e2ae394fdf09f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:49:20.766731 31565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 418cf4276bc44e41803e2ae394fdf09f [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:49:20.769186 31565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 418cf4276bc44e41803e2ae394fdf09f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "418cf4276bc44e41803e2ae394fdf09f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40283 } }
I20260926 16:49:20.769726 31565 leader_election.cc:302] T 00000000000000000000000000000000 P 418cf4276bc44e41803e2ae394fdf09f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 418cf4276bc44e41803e2ae394fdf09f; no voters:
I20260926 16:49:20.770391 31565 leader_election.cc:288] T 00000000000000000000000000000000 P 418cf4276bc44e41803e2ae394fdf09f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:49:20.770828 31575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 418cf4276bc44e41803e2ae394fdf09f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:49:20.772486 31575 raft_consensus.cc:686] T 00000000000000000000000000000000 P 418cf4276bc44e41803e2ae394fdf09f [term 1 LEADER]: Becoming Leader. State: Replica: 418cf4276bc44e41803e2ae394fdf09f, State: Running, Role: LEADER
I20260926 16:49:20.772924 31565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 418cf4276bc44e41803e2ae394fdf09f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:49:20.773327 31575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 418cf4276bc44e41803e2ae394fdf09f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "418cf4276bc44e41803e2ae394fdf09f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40283 } }
I20260926 16:49:20.777848 31577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 418cf4276bc44e41803e2ae394fdf09f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 418cf4276bc44e41803e2ae394fdf09f. Latest consensus state: current_term: 1 leader_uuid: "418cf4276bc44e41803e2ae394fdf09f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "418cf4276bc44e41803e2ae394fdf09f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40283 } } }
I20260926 16:49:20.777812 31576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 418cf4276bc44e41803e2ae394fdf09f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "418cf4276bc44e41803e2ae394fdf09f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "418cf4276bc44e41803e2ae394fdf09f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40283 } } }
I20260926 16:49:20.778570 31577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 418cf4276bc44e41803e2ae394fdf09f [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:20.778652 31576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 418cf4276bc44e41803e2ae394fdf09f [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:20.783373 31584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:49:20.790601 31584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:49:20.801316 31584 catalog_manager.cc:1347] Generated new cluster ID: 1a0fcadffc5a4228a895f57d1aa67082
I20260926 16:49:20.801476 31584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:49:20.819532 31584 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:49:20.820269 31584 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:49:20.835700 31584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 418cf4276bc44e41803e2ae394fdf09f: Generated new TSK 0
I20260926 16:49:20.837232 31584 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:49:20.854578 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 31498
W20260926 16:49:20.856122 31560 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.144:45867
W20260926 16:49:20.857087 10171 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 31498: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:49:20.857307 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 31498
2026-09-26T16:49:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2100 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:49:20 dist-test-slave-wt2j krb5kdc[31596](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:49:20 dist-test-slave-wt2j krb5kdc[31596](info): set up 2 sockets
Sep 26 16:49:20 dist-test-slave-wt2j krb5kdc[31596](info): commencing operation
krb5kdc: starting...
W20260926 16:49:22.037986 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.001s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:49:22 dist-test-slave-wt2j krb5kdc[31596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:49:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:49:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:49:22.352532 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39821
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:49:22.790587 31609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:49:22.790912 31609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:49:22.791008 31609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:49:22.798485 31609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:49:22.798622 31609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:49:22.798671 31609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:49:22.798724 31609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:49:22.798763 31609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:49:22.809284 31609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39821
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:49:22.810667 31609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:49:22.812012 31609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:49:22.826450 31614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:22.826608 31615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:22.827057 31617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:49:22.827713 31609 server_base.cc:1034] running on GCE node
I20260926 16:49:22.828897 31609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:49:22.831136 31609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:49:22.832778 31609 hybrid_clock.cc:648] HybridClock initialized: now 1790441362832651 us; error 115 us; skew 500 ppm
I20260926 16:49:22.833392 31609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:49:22 dist-test-slave-wt2j krb5kdc[31596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:49:22.837867 31609 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:49:22.840144 31609 webserver.cc:460] Webserver started at http://127.0.0.1:40681/ using document root <none> and password file <none>
I20260926 16:49:22.841125 31609 fs_manager.cc:359] Metadata directory not provided
I20260926 16:49:22.841369 31609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:49:22.841811 31609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:49:22.846009 31609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "e09b821711bf4bb39a9ff409c0d85b2c"
format_stamp: "Formatted at 2026-09-26 16:49:22 on dist-test-slave-wt2j"
server_key: "bcdb6b32ae06dccd262b64a524a10db5"
server_key_iv: "95d223618e3265597b8e78489399dc06"
server_key_version: "encryptionkey@0"
I20260926 16:49:22.847159 31609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "e09b821711bf4bb39a9ff409c0d85b2c"
format_stamp: "Formatted at 2026-09-26 16:49:22 on dist-test-slave-wt2j"
server_key: "bcdb6b32ae06dccd262b64a524a10db5"
server_key_iv: "95d223618e3265597b8e78489399dc06"
server_key_version: "encryptionkey@0"
I20260926 16:49:22.854971 31609 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260926 16:49:22.861048 31624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:49:22.863011 31609 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260926 16:49:22.863386 31609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "e09b821711bf4bb39a9ff409c0d85b2c"
format_stamp: "Formatted at 2026-09-26 16:49:22 on dist-test-slave-wt2j"
server_key: "bcdb6b32ae06dccd262b64a524a10db5"
server_key_iv: "95d223618e3265597b8e78489399dc06"
server_key_version: "encryptionkey@0"
I20260926 16:49:22.863768 31609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:49:22.896782 31609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:49:22.902280 31609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:49:22.902628 31609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:49:22.926308 31609 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39821
I20260926 16:49:22.926349 31675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39821 every 8 connection(s)
I20260926 16:49:22.928144 31609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:49:22.929275 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 31609
I20260926 16:49:22.929531 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:49:22.930135 10171 external_mini_cluster.cc:1428] Setting key 96f14118842cf6e70c014e8f0e8b279f
I20260926 16:49:22.933284 31676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 16:49:22 dist-test-slave-wt2j krb5kdc[31596](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 16:49:22.948073 31676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e09b821711bf4bb39a9ff409c0d85b2c: Bootstrap starting.
I20260926 16:49:22.951930 31676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e09b821711bf4bb39a9ff409c0d85b2c: Neither blocks nor log segments found. Creating new log.
I20260926 16:49:22.953203 31676 log.cc:824] T 00000000000000000000000000000000 P e09b821711bf4bb39a9ff409c0d85b2c: Log is configured to *not* fsync() on all Append() calls
I20260926 16:49:22.957044 31676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e09b821711bf4bb39a9ff409c0d85b2c: No bootstrap required, opened a new log
I20260926 16:49:22.958194 31679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:22.932376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:53911 (local address 127.0.0.1:39821)
0926 16:49:22.933384 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:22.933413 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:22.933487 (+ 74us) server_negotiation.cc:407] Connection header received
0926 16:49:22.935449 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:22.935476 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:22.936081 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:22.936557 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:22.938484 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:22.940204 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:22.942130 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:22.942976 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:22.943151 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:22.948632 (+ 5481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:22.948647 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:22.948669 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:22.948730 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:22.952590 (+ 3860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:22.953646 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:22.953654 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:22.953662 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:22.953990 (+ 328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:22.954527 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:22.954538 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:22.954543 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:22.955387 (+ 844us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:22.955735 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:22.956290 (+ 555us) server_negotiation.cc:299] Negotiation successful
0926 16:49:22.956741 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":438,"threads_started":1}
I20260926 16:49:22.963645 10171 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:49:22.966061 31676 raft_consensus.cc:348] T 00000000000000000000000000000000 P e09b821711bf4bb39a9ff409c0d85b2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09b821711bf4bb39a9ff409c0d85b2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39821 } }
I20260926 16:49:22.966496 31676 raft_consensus.cc:374] T 00000000000000000000000000000000 P e09b821711bf4bb39a9ff409c0d85b2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:49:22.966610 31676 raft_consensus.cc:729] T 00000000000000000000000000000000 P e09b821711bf4bb39a9ff409c0d85b2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e09b821711bf4bb39a9ff409c0d85b2c, State: Initialized, Role: FOLLOWER
I20260926 16:49:22.967396 31676 consensus_queue.cc:260] T 00000000000000000000000000000000 P e09b821711bf4bb39a9ff409c0d85b2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09b821711bf4bb39a9ff409c0d85b2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39821 } }
I20260926 16:49:22.967685 31676 raft_consensus.cc:388] T 00000000000000000000000000000000 P e09b821711bf4bb39a9ff409c0d85b2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:49:22.967855 31676 raft_consensus.cc:482] T 00000000000000000000000000000000 P e09b821711bf4bb39a9ff409c0d85b2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:49:22.968057 31676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e09b821711bf4bb39a9ff409c0d85b2c [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:49:22.971014 31676 raft_consensus.cc:504] T 00000000000000000000000000000000 P e09b821711bf4bb39a9ff409c0d85b2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09b821711bf4bb39a9ff409c0d85b2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39821 } }
I20260926 16:49:22.971668 31676 leader_election.cc:302] T 00000000000000000000000000000000 P e09b821711bf4bb39a9ff409c0d85b2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e09b821711bf4bb39a9ff409c0d85b2c; no voters:
I20260926 16:49:22.972369 31676 leader_election.cc:288] T 00000000000000000000000000000000 P e09b821711bf4bb39a9ff409c0d85b2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:49:22.972635 31686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e09b821711bf4bb39a9ff409c0d85b2c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:49:22.974606 31676 sys_catalog.cc:564] T 00000000000000000000000000000000 P e09b821711bf4bb39a9ff409c0d85b2c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:49:22.978648 31686 raft_consensus.cc:686] T 00000000000000000000000000000000 P e09b821711bf4bb39a9ff409c0d85b2c [term 1 LEADER]: Becoming Leader. State: Replica: e09b821711bf4bb39a9ff409c0d85b2c, State: Running, Role: LEADER
I20260926 16:49:22.979789 31686 consensus_queue.cc:237] T 00000000000000000000000000000000 P e09b821711bf4bb39a9ff409c0d85b2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09b821711bf4bb39a9ff409c0d85b2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39821 } }
I20260926 16:49:22.985519 31687 sys_catalog.cc:455] T 00000000000000000000000000000000 P e09b821711bf4bb39a9ff409c0d85b2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e09b821711bf4bb39a9ff409c0d85b2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09b821711bf4bb39a9ff409c0d85b2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39821 } } }
I20260926 16:49:22.986038 31687 sys_catalog.cc:458] T 00000000000000000000000000000000 P e09b821711bf4bb39a9ff409c0d85b2c [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:22.986891 31688 sys_catalog.cc:455] T 00000000000000000000000000000000 P e09b821711bf4bb39a9ff409c0d85b2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e09b821711bf4bb39a9ff409c0d85b2c. Latest consensus state: current_term: 1 leader_uuid: "e09b821711bf4bb39a9ff409c0d85b2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09b821711bf4bb39a9ff409c0d85b2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39821 } } }
I20260926 16:49:22.987309 31688 sys_catalog.cc:458] T 00000000000000000000000000000000 P e09b821711bf4bb39a9ff409c0d85b2c [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:22.996263 31679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:22.969555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:56649 (local address 127.0.0.1:39821)
0926 16:49:22.969816 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:22.969829 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:22.970445 (+ 616us) server_negotiation.cc:407] Connection header received
0926 16:49:22.970697 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:22.970711 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:22.970803 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:22.970938 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:22.973296 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:22.975208 (+ 1912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:22.978137 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:22.979130 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:22.979289 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:22.983687 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:22.983699 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:22.983704 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:22.983762 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:22.989470 (+ 5708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:22.991025 (+ 1555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:22.991036 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:22.991042 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:22.991712 (+ 670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:22.992499 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:22.992507 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:22.992513 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:22.993068 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:22.995058 (+ 1990us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:22.995325 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 16:49:22.995654 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260926 16:49:22.999166 31700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:49:23.006853 31700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:49:23.018824 31700 catalog_manager.cc:1347] Generated new cluster ID: 9c2990f124184ff88fa7552c87d2315e
I20260926 16:49:23.019044 31700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:49:23.037954 31700 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:49:23.039738 31700 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:49:23.047209 31700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e09b821711bf4bb39a9ff409c0d85b2c: Generated new TSK 0
I20260926 16:49:23.048519 31700 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:49:23.074702 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 31609
I20260926 16:49:23.077055 31671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:49:23.179096 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 31609
W20260926 16:49:23.202114 31594 connection.cc:535] client connection to 127.0.0.1:39821 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39821): Connection reset by peer (error 104)
2026-09-26T16:49:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:49:23 dist-test-slave-wt2j krb5kdc[31708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:49:23 dist-test-slave-wt2j krb5kdc[31708](info): set up 2 sockets
Sep 26 16:49:23 dist-test-slave-wt2j krb5kdc[31708](info): commencing operation
krb5kdc: starting...
W20260926 16:49:24.349641 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:49:24 dist-test-slave-wt2j krb5kdc[31708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:49:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:49:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:49:24.664283 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43831
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:49:25.031890 31721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:49:25.032212 31721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:49:25.032308 31721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:49:25.039673 31721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:49:25.039811 31721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:49:25.039861 31721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:49:25.039917 31721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:49:25.039968 31721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:49:25.050495 31721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43831
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:49:25.051911 31721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:49:25.053216 31721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:49:25.066984 31727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:25.067034 31729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:25.067101 31726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:49:25.068109 31721 server_base.cc:1034] running on GCE node
I20260926 16:49:25.069015 31721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:49:25.070633 31721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:49:25.071882 31721 hybrid_clock.cc:648] HybridClock initialized: now 1790441365071819 us; error 55 us; skew 500 ppm
I20260926 16:49:25.072274 31721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:49:25 dist-test-slave-wt2j krb5kdc[31708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:49:25.076121 31721 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:49:25.078133 31721 webserver.cc:460] Webserver started at http://127.0.0.1:38183/ using document root <none> and password file <none>
I20260926 16:49:25.078951 31721 fs_manager.cc:359] Metadata directory not provided
I20260926 16:49:25.079118 31721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:49:25.079447 31721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:49:25.082647 31721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "54c2d8dd58d24a66a26542e135ef1138"
format_stamp: "Formatted at 2026-09-26 16:49:25 on dist-test-slave-wt2j"
server_key: "7305306ca009df8d71524054181141a8"
server_key_iv: "cb4c07b42062852850b9c587d4045abf"
server_key_version: "encryptionkey@0"
I20260926 16:49:25.083472 31721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "54c2d8dd58d24a66a26542e135ef1138"
format_stamp: "Formatted at 2026-09-26 16:49:25 on dist-test-slave-wt2j"
server_key: "7305306ca009df8d71524054181141a8"
server_key_iv: "cb4c07b42062852850b9c587d4045abf"
server_key_version: "encryptionkey@0"
I20260926 16:49:25.089697 31721 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260926 16:49:25.094153 31736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:49:25.096637 31721 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260926 16:49:25.096930 31721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "54c2d8dd58d24a66a26542e135ef1138"
format_stamp: "Formatted at 2026-09-26 16:49:25 on dist-test-slave-wt2j"
server_key: "7305306ca009df8d71524054181141a8"
server_key_iv: "cb4c07b42062852850b9c587d4045abf"
server_key_version: "encryptionkey@0"
I20260926 16:49:25.097206 31721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:49:25.114168 31721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:49:25.119246 31721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:49:25.119613 31721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:49:25.142155 31721 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43831
I20260926 16:49:25.142346 31787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43831 every 8 connection(s)
I20260926 16:49:25.143734 31721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:49:25.148245 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 31721
I20260926 16:49:25.148408 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:49:25.149087 10171 external_mini_cluster.cc:1428] Setting key 592f1a468a23f5a75b786a7e323b6b82
I20260926 16:49:25.149576 31788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:49:25.169353 31791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:25.151499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:54999 (local address 127.0.0.1:43831)
0926 16:49:25.152585 (+ 1086us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:25.152605 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:25.152675 (+ 70us) server_negotiation.cc:407] Connection header received
0926 16:49:25.156044 (+ 3369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:25.156088 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:25.156933 (+ 845us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:25.157649 (+ 716us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:25.158415 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:25.160873 (+ 2458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:25.162490 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:25.163792 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:25.164040 (+ 248us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:25.165551 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:25.165564 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:25.165596 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:25.166626 (+ 1030us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:25.166758 (+ 132us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:25.166812 (+ 54us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:25.167001 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:25.167374 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 16:49:25.167688 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":772,"thread_start_us":405,"threads_started":1}
I20260926 16:49:25.169478 31788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54c2d8dd58d24a66a26542e135ef1138: Bootstrap starting.
I20260926 16:49:25.173939 31788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54c2d8dd58d24a66a26542e135ef1138: Neither blocks nor log segments found. Creating new log.
I20260926 16:49:25.174786 10171 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:49:25.175644 31788 log.cc:824] T 00000000000000000000000000000000 P 54c2d8dd58d24a66a26542e135ef1138: Log is configured to *not* fsync() on all Append() calls
I20260926 16:49:25.179836 31788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54c2d8dd58d24a66a26542e135ef1138: No bootstrap required, opened a new log
I20260926 16:49:25.188349 31791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:25.180619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:48479 (local address 127.0.0.1:43831)
0926 16:49:25.180867 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:25.180876 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:25.181140 (+ 264us) server_negotiation.cc:407] Connection header received
0926 16:49:25.181333 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:25.181339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:25.181442 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:25.181601 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:25.182149 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:25.183409 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:25.184996 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:25.185746 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:25.185905 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:25.187155 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:25.187162 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:25.187168 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:25.187202 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:25.187271 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:25.187295 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:25.187434 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:25.187863 (+ 429us) server_negotiation.cc:299] Negotiation successful
0926 16:49:25.187942 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 16:49:25.191006 31788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54c2d8dd58d24a66a26542e135ef1138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c2d8dd58d24a66a26542e135ef1138" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43831 } }
I20260926 16:49:25.191473 31788 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54c2d8dd58d24a66a26542e135ef1138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:49:25.191635 31788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54c2d8dd58d24a66a26542e135ef1138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54c2d8dd58d24a66a26542e135ef1138, State: Initialized, Role: FOLLOWER
I20260926 16:49:25.192345 31788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54c2d8dd58d24a66a26542e135ef1138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c2d8dd58d24a66a26542e135ef1138" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43831 } }
I20260926 16:49:25.192623 31788 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54c2d8dd58d24a66a26542e135ef1138 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:49:25.192792 31788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54c2d8dd58d24a66a26542e135ef1138 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:49:25.192989 31788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54c2d8dd58d24a66a26542e135ef1138 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:49:25.195683 31788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54c2d8dd58d24a66a26542e135ef1138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c2d8dd58d24a66a26542e135ef1138" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43831 } }
I20260926 16:49:25.196412 31788 leader_election.cc:302] T 00000000000000000000000000000000 P 54c2d8dd58d24a66a26542e135ef1138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54c2d8dd58d24a66a26542e135ef1138; no voters:
I20260926 16:49:25.197185 31788 leader_election.cc:288] T 00000000000000000000000000000000 P 54c2d8dd58d24a66a26542e135ef1138 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:49:25.197461 31798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54c2d8dd58d24a66a26542e135ef1138 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:49:25.198782 31798 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54c2d8dd58d24a66a26542e135ef1138 [term 1 LEADER]: Becoming Leader. State: Replica: 54c2d8dd58d24a66a26542e135ef1138, State: Running, Role: LEADER
I20260926 16:49:25.199508 31798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54c2d8dd58d24a66a26542e135ef1138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c2d8dd58d24a66a26542e135ef1138" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43831 } }
I20260926 16:49:25.200328 31788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54c2d8dd58d24a66a26542e135ef1138 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:49:25.203529 31799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54c2d8dd58d24a66a26542e135ef1138 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54c2d8dd58d24a66a26542e135ef1138" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c2d8dd58d24a66a26542e135ef1138" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43831 } } }
I20260926 16:49:25.203799 31800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54c2d8dd58d24a66a26542e135ef1138 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54c2d8dd58d24a66a26542e135ef1138. Latest consensus state: current_term: 1 leader_uuid: "54c2d8dd58d24a66a26542e135ef1138" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c2d8dd58d24a66a26542e135ef1138" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43831 } } }
I20260926 16:49:25.204149 31799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54c2d8dd58d24a66a26542e135ef1138 [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:25.204252 31800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54c2d8dd58d24a66a26542e135ef1138 [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:25.206601 31805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:49:25.216050 31805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:49:25.228641 31805 catalog_manager.cc:1347] Generated new cluster ID: 37dc3d61930443b581a5508ace3f5fff
I20260926 16:49:25.228830 31805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:49:25.238935 31805 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:49:25.241364 31805 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:49:25.252928 31805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54c2d8dd58d24a66a26542e135ef1138: Generated new TSK 0
I20260926 16:49:25.254563 31805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:49:25.271402 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 31721
W20260926 16:49:25.273160 31783 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.144:54999
W20260926 16:49:25.274029 10171 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 31721: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:49:25.274231 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 31721
W20260926 16:49:25.292263 31706 connection.cc:535] client connection to 127.0.0.1:43831 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43831): Connection reset by peer (error 104)
2026-09-26T16:49:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2087 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:49:25 dist-test-slave-wt2j krb5kdc[31819](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:49:25 dist-test-slave-wt2j krb5kdc[31819](info): set up 2 sockets
Sep 26 16:49:25 dist-test-slave-wt2j krb5kdc[31819](info): commencing operation
krb5kdc: starting...
W20260926 16:49:26.425410 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:49:26 dist-test-slave-wt2j krb5kdc[31819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:49:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:49:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 16:49:26.744539 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44477
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:49:27.087783 31832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:49:27.088172 31832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:49:27.088358 31832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:49:27.096441 31832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:49:27.096640 31832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:49:27.096761 31832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:49:27.096882 31832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:49:27.096980 31832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:49:27.107615 31832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44477
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440988577176-10171-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:49:27.109086 31832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:49:27.110468 31832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:49:27.125263 31838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:27.125370 31837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:27.125396 31840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:49:27.126434 31832 server_base.cc:1034] running on GCE node
I20260926 16:49:27.127434 31832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:49:27.129144 31832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:49:27.130436 31832 hybrid_clock.cc:648] HybridClock initialized: now 1790441367130385 us; error 47 us; skew 500 ppm
I20260926 16:49:27.130856 31832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:49:27 dist-test-slave-wt2j krb5kdc[31819](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:49:27.134740 31832 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 16:49:27.136651 31832 webserver.cc:460] Webserver started at http://127.0.0.1:44907/ using document root <none> and password file <none>
I20260926 16:49:27.137430 31832 fs_manager.cc:359] Metadata directory not provided
I20260926 16:49:27.137601 31832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:49:27.137944 31832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:49:27.141386 31832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "518e8d1c99f44c43a787b76a397e08f4"
format_stamp: "Formatted at 2026-09-26 16:49:27 on dist-test-slave-wt2j"
server_key: "05ded09cc4fb7939364791406459a863"
server_key_iv: "4bf8b51bdb2bffcde6b9a578cda3d76d"
server_key_version: "encryptionkey@0"
I20260926 16:49:27.142246 31832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "518e8d1c99f44c43a787b76a397e08f4"
format_stamp: "Formatted at 2026-09-26 16:49:27 on dist-test-slave-wt2j"
server_key: "05ded09cc4fb7939364791406459a863"
server_key_iv: "4bf8b51bdb2bffcde6b9a578cda3d76d"
server_key_version: "encryptionkey@0"
I20260926 16:49:27.148691 31832 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260926 16:49:27.153239 31847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:49:27.155139 31832 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260926 16:49:27.155398 31832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "518e8d1c99f44c43a787b76a397e08f4"
format_stamp: "Formatted at 2026-09-26 16:49:27 on dist-test-slave-wt2j"
server_key: "05ded09cc4fb7939364791406459a863"
server_key_iv: "4bf8b51bdb2bffcde6b9a578cda3d76d"
server_key_version: "encryptionkey@0"
I20260926 16:49:27.155694 31832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:49:27.162719 31832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:49:27.168006 31832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:49:27.168342 31832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:49:27.190845 31832 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44477
I20260926 16:49:27.190907 31898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44477 every 8 connection(s)
I20260926 16:49:27.192830 31832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:49:27.196187 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 31832
I20260926 16:49:27.196502 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:49:27.197067 10171 external_mini_cluster.cc:1428] Setting key 2ff4fab6eed153131c6dbb6a4e738249
I20260926 16:49:27.197576 31899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:49:27.207864 31902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:27.199683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:36209 (local address 127.0.0.1:44477)
0926 16:49:27.200712 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:27.200732 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:27.200796 (+ 64us) server_negotiation.cc:407] Connection header received
0926 16:49:27.203226 (+ 2430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:27.203258 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:27.203810 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:27.204352 (+ 542us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:27.205639 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:27.205651 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:27.205676 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:27.205704 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:27.205793 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:27.205835 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:27.206134 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:27.206285 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 16:49:27.206519 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":388,"threads_started":1}
I20260926 16:49:27.213490 10171 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:49:27.214140 31899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 518e8d1c99f44c43a787b76a397e08f4: Bootstrap starting.
I20260926 16:49:27.217875 31899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 518e8d1c99f44c43a787b76a397e08f4: Neither blocks nor log segments found. Creating new log.
I20260926 16:49:27.219061 31899 log.cc:824] T 00000000000000000000000000000000 P 518e8d1c99f44c43a787b76a397e08f4: Log is configured to *not* fsync() on all Append() calls
I20260926 16:49:27.222615 31902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:27.219383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.144:55339 (local address 127.0.0.1:44477)
0926 16:49:27.219628 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:27.219638 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:27.220068 (+ 430us) server_negotiation.cc:407] Connection header received
0926 16:49:27.220263 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:27.220270 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:27.220354 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:27.220545 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:27.221792 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:27.221798 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:27.221803 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:27.221822 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:27.221871 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:27.221894 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:27.222043 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:27.222232 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 16:49:27.222303 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 16:49:27.222755 31899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 518e8d1c99f44c43a787b76a397e08f4: No bootstrap required, opened a new log
I20260926 16:49:27.231076 31899 raft_consensus.cc:348] T 00000000000000000000000000000000 P 518e8d1c99f44c43a787b76a397e08f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518e8d1c99f44c43a787b76a397e08f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44477 } }
I20260926 16:49:27.231478 31899 raft_consensus.cc:374] T 00000000000000000000000000000000 P 518e8d1c99f44c43a787b76a397e08f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:49:27.231628 31899 raft_consensus.cc:729] T 00000000000000000000000000000000 P 518e8d1c99f44c43a787b76a397e08f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 518e8d1c99f44c43a787b76a397e08f4, State: Initialized, Role: FOLLOWER
I20260926 16:49:27.232302 31899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 518e8d1c99f44c43a787b76a397e08f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518e8d1c99f44c43a787b76a397e08f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44477 } }
I20260926 16:49:27.232573 31899 raft_consensus.cc:388] T 00000000000000000000000000000000 P 518e8d1c99f44c43a787b76a397e08f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:49:27.232759 31899 raft_consensus.cc:482] T 00000000000000000000000000000000 P 518e8d1c99f44c43a787b76a397e08f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:49:27.232950 31899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 518e8d1c99f44c43a787b76a397e08f4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:49:27.235530 31899 raft_consensus.cc:504] T 00000000000000000000000000000000 P 518e8d1c99f44c43a787b76a397e08f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518e8d1c99f44c43a787b76a397e08f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44477 } }
I20260926 16:49:27.236196 31899 leader_election.cc:302] T 00000000000000000000000000000000 P 518e8d1c99f44c43a787b76a397e08f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 518e8d1c99f44c43a787b76a397e08f4; no voters:
I20260926 16:49:27.236979 31899 leader_election.cc:288] T 00000000000000000000000000000000 P 518e8d1c99f44c43a787b76a397e08f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:49:27.237265 31909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 518e8d1c99f44c43a787b76a397e08f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:49:27.238423 31909 raft_consensus.cc:686] T 00000000000000000000000000000000 P 518e8d1c99f44c43a787b76a397e08f4 [term 1 LEADER]: Becoming Leader. State: Replica: 518e8d1c99f44c43a787b76a397e08f4, State: Running, Role: LEADER
I20260926 16:49:27.239408 31899 sys_catalog.cc:564] T 00000000000000000000000000000000 P 518e8d1c99f44c43a787b76a397e08f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:49:27.239280 31909 consensus_queue.cc:237] T 00000000000000000000000000000000 P 518e8d1c99f44c43a787b76a397e08f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518e8d1c99f44c43a787b76a397e08f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44477 } }
I20260926 16:49:27.244884 31910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 518e8d1c99f44c43a787b76a397e08f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "518e8d1c99f44c43a787b76a397e08f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518e8d1c99f44c43a787b76a397e08f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44477 } } }
I20260926 16:49:27.244978 31911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 518e8d1c99f44c43a787b76a397e08f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 518e8d1c99f44c43a787b76a397e08f4. Latest consensus state: current_term: 1 leader_uuid: "518e8d1c99f44c43a787b76a397e08f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518e8d1c99f44c43a787b76a397e08f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44477 } } }
I20260926 16:49:27.245365 31910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 518e8d1c99f44c43a787b76a397e08f4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:27.246491 31911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 518e8d1c99f44c43a787b76a397e08f4 [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:27.247495 31918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:49:27.264017 31918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:49:27.275791 31918 catalog_manager.cc:1347] Generated new cluster ID: 88054795331a44279fa1b8a5bfd27bf8
I20260926 16:49:27.275975 31918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:49:27.293202 31918 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:49:27.293885 31918 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:49:27.308959 31918 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 518e8d1c99f44c43a787b76a397e08f4: Generated new TSK 0
I20260926 16:49:27.310267 31918 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:49:27.324636 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 31832
W20260926 16:49:27.326041 31894 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.144:36209
W20260926 16:49:27.326978 10171 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 31832: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:49:27.327116 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 31832
2026-09-26T16:49:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2053 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:49:27 dist-test-slave-wt2j krb5kdc[31930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:49:27 dist-test-slave-wt2j krb5kdc[31930](info): set up 2 sockets
Sep 26 16:49:27 dist-test-slave-wt2j krb5kdc[31930](info): commencing operation
krb5kdc: starting...
W20260926 16:49:28.493484 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.128s user 0.004s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:49:28 dist-test-slave-wt2j krb5kdc[31930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:49:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:49:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:49:28.820101 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:35581
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:37497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:35581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:49:29.169037 31943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:49:29.169391 31943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:49:29.169561 31943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:49:29.177341 31943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:49:29.177538 31943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:49:29.177654 31943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:49:29.177788 31943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:49:29.177871 31943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:49:29.188318 31943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:37497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:35581
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:35581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:49:29.189847 31943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:49:29.191205 31943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:49:29.204485 31949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:29.204928 31951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:29.205565 31948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:49:29.206168 31943 server_base.cc:1034] running on GCE node
I20260926 16:49:29.207093 31943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:49:29.208755 31943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:49:29.210057 31943 hybrid_clock.cc:648] HybridClock initialized: now 1790441369209996 us; error 44 us; skew 500 ppm
I20260926 16:49:29.210511 31943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:49:29 dist-test-slave-wt2j krb5kdc[31930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441369, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:49:29.219798 31943 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:49:29.221665 31943 webserver.cc:460] Webserver started at http://127.9.238.254:43189/ using document root <none> and password file <none>
I20260926 16:49:29.222425 31943 fs_manager.cc:359] Metadata directory not provided
I20260926 16:49:29.222587 31943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:49:29.222908 31943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:49:29.226121 31943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "b9847ad4dae840909d32f19be8f1298e"
format_stamp: "Formatted at 2026-09-26 16:49:29 on dist-test-slave-wt2j"
server_key: "a9f1725b089017c881cadb2ed3d39874"
server_key_iv: "723f918d9497440daf4fca538154ea6f"
server_key_version: "encryptionkey@0"
I20260926 16:49:29.226928 31943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "b9847ad4dae840909d32f19be8f1298e"
format_stamp: "Formatted at 2026-09-26 16:49:29 on dist-test-slave-wt2j"
server_key: "a9f1725b089017c881cadb2ed3d39874"
server_key_iv: "723f918d9497440daf4fca538154ea6f"
server_key_version: "encryptionkey@0"
I20260926 16:49:29.233006 31943 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 16:49:29.237345 31958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:49:29.239089 31943 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:49:29.239316 31943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "b9847ad4dae840909d32f19be8f1298e"
format_stamp: "Formatted at 2026-09-26 16:49:29 on dist-test-slave-wt2j"
server_key: "a9f1725b089017c881cadb2ed3d39874"
server_key_iv: "723f918d9497440daf4fca538154ea6f"
server_key_version: "encryptionkey@0"
I20260926 16:49:29.239538 31943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:49:29.268697 31943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:49:29.273699 31943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:49:29.274039 31943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:49:29.296911 31943 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:35581
I20260926 16:49:29.296962 32009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:35581 every 8 connection(s)
I20260926 16:49:29.298455 31943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:49:29.303705 32010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:49:29.304116 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 31943
I20260926 16:49:29.304318 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:49:29.304992 10171 external_mini_cluster.cc:1428] Setting key 83db587122ba3de2abe0f104f9f9b25e
I20260926 16:49:29.321866 32010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9847ad4dae840909d32f19be8f1298e: Bootstrap starting.
Sep 26 16:49:29 dist-test-slave-wt2j krb5kdc[31930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:49:29.326596 32010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9847ad4dae840909d32f19be8f1298e: Neither blocks nor log segments found. Creating new log.
I20260926 16:49:29.328222 32010 log.cc:824] T 00000000000000000000000000000000 P b9847ad4dae840909d32f19be8f1298e: Log is configured to *not* fsync() on all Append() calls
I20260926 16:49:29.332661 32010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9847ad4dae840909d32f19be8f1298e: No bootstrap required, opened a new log
I20260926 16:49:29.335644 32013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:29.307838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34243 (local address 127.9.238.254:35581)
0926 16:49:29.308810 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:29.308823 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:29.308874 (+ 51us) server_negotiation.cc:407] Connection header received
0926 16:49:29.311632 (+ 2758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:29.311676 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:29.312203 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:29.312596 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:29.314897 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:29.316441 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:29.318527 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:29.319357 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:29.319519 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:29.325676 (+ 6157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:29.325686 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:29.325709 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:29.325761 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:29.330221 (+ 4460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:29.331506 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:29.331515 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:29.331531 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:29.331794 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:29.332341 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:29.332345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:29.332347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:29.333003 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:29.333264 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:29.333900 (+ 636us) server_negotiation.cc:299] Negotiation successful
0926 16:49:29.334310 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":446,"threads_started":1}
I20260926 16:49:29.340225 10171 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:49:29.343125 32010 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9847ad4dae840909d32f19be8f1298e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9847ad4dae840909d32f19be8f1298e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35581 } }
I20260926 16:49:29.343847 32010 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9847ad4dae840909d32f19be8f1298e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:49:29.344076 32010 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9847ad4dae840909d32f19be8f1298e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9847ad4dae840909d32f19be8f1298e, State: Initialized, Role: FOLLOWER
I20260926 16:49:29.345074 32010 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9847ad4dae840909d32f19be8f1298e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9847ad4dae840909d32f19be8f1298e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35581 } }
I20260926 16:49:29.345424 32010 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9847ad4dae840909d32f19be8f1298e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:49:29.345598 32010 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9847ad4dae840909d32f19be8f1298e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:49:29.345791 32010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9847ad4dae840909d32f19be8f1298e [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:49:29.349044 32010 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9847ad4dae840909d32f19be8f1298e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9847ad4dae840909d32f19be8f1298e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35581 } }
I20260926 16:49:29.349819 32010 leader_election.cc:302] T 00000000000000000000000000000000 P b9847ad4dae840909d32f19be8f1298e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9847ad4dae840909d32f19be8f1298e; no voters:
I20260926 16:49:29.350642 32010 leader_election.cc:288] T 00000000000000000000000000000000 P b9847ad4dae840909d32f19be8f1298e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:49:29.350889 32020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9847ad4dae840909d32f19be8f1298e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:49:29.352993 32020 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9847ad4dae840909d32f19be8f1298e [term 1 LEADER]: Becoming Leader. State: Replica: b9847ad4dae840909d32f19be8f1298e, State: Running, Role: LEADER
I20260926 16:49:29.353842 32020 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9847ad4dae840909d32f19be8f1298e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9847ad4dae840909d32f19be8f1298e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35581 } }
I20260926 16:49:29.354218 32010 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9847ad4dae840909d32f19be8f1298e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:49:29.358481 32021 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9847ad4dae840909d32f19be8f1298e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9847ad4dae840909d32f19be8f1298e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9847ad4dae840909d32f19be8f1298e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35581 } } }
I20260926 16:49:29.358983 32021 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9847ad4dae840909d32f19be8f1298e [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:29.359680 32022 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9847ad4dae840909d32f19be8f1298e [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9847ad4dae840909d32f19be8f1298e. Latest consensus state: current_term: 1 leader_uuid: "b9847ad4dae840909d32f19be8f1298e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9847ad4dae840909d32f19be8f1298e" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 35581 } } }
I20260926 16:49:29.360069 32022 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9847ad4dae840909d32f19be8f1298e [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:29.360890 32031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:49:29.366441 32031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:49:29.369601 32013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:29.346657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33233 (local address 127.9.238.254:35581)
0926 16:49:29.346946 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:29.346956 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:29.347475 (+ 519us) server_negotiation.cc:407] Connection header received
0926 16:49:29.347745 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:29.347751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:29.347857 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:29.348018 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:29.349654 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:29.350945 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:29.352873 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:29.353639 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:29.353802 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:29.357659 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:29.357666 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:29.357672 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:29.357734 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:29.363317 (+ 5583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:29.364987 (+ 1670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:29.364993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:29.364999 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:29.365166 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:29.365921 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:29.365929 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:29.365935 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:29.366581 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:29.368363 (+ 1782us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:29.368651 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 16:49:29.368972 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 16:49:29.385035 32031 catalog_manager.cc:1347] Generated new cluster ID: 03c6e774d08c48ffa059545c632d306d
I20260926 16:49:29.385425 32031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:49:29.406276 32031 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:49:29.408064 32031 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:49:29.419657 32031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9847ad4dae840909d32f19be8f1298e: Generated new TSK 0
I20260926 16:49:29.420869 32031 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:49:29.429978 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 31943
I20260926 16:49:29.431983 32005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:49:29.537002 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 31943
W20260926 16:49:29.555524 31928 connection.cc:535] client connection to 127.9.238.254:35581 recv error: Network error: failed to read from TLS socket (remote: 127.9.238.254:35581): Connection reset by peer (error 104)
2026-09-26T16:49:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2209 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:49:29 dist-test-slave-wt2j krb5kdc[32042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:49:29 dist-test-slave-wt2j krb5kdc[32042](info): set up 2 sockets
Sep 26 16:49:29 dist-test-slave-wt2j krb5kdc[32042](info): commencing operation
krb5kdc: starting...
W20260926 16:49:30.685807 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:49:30 dist-test-slave-wt2j krb5kdc[32042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:49:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:49:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:49:31.015946 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:41281
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:45153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:41281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:49:31.355235 32055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:49:31.355645 32055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:49:31.355814 32055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:49:31.363359 32055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:49:31.363538 32055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:49:31.363674 32055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:49:31.363806 32055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:49:31.363889 32055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:49:31.374114 32055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:45153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:41281
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:41281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:49:31.375664 32055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:49:31.377033 32055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:49:31.390914 32060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:31.391108 32061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:31.391150 32063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:49:31.392343 32055 server_base.cc:1034] running on GCE node
I20260926 16:49:31.393319 32055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:49:31.394843 32055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:49:31.396090 32055 hybrid_clock.cc:648] HybridClock initialized: now 1790441371396027 us; error 56 us; skew 500 ppm
I20260926 16:49:31.396477 32055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:49:31 dist-test-slave-wt2j krb5kdc[32042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441371, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:49:31.400785 32055 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:49:31.402752 32055 webserver.cc:460] Webserver started at http://127.9.238.254:33891/ using document root <none> and password file <none>
I20260926 16:49:31.403731 32055 fs_manager.cc:359] Metadata directory not provided
I20260926 16:49:31.403868 32055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:49:31.404209 32055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:49:31.408389 32055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "2e07ee7c976749e9afb5290d785f326c"
format_stamp: "Formatted at 2026-09-26 16:49:31 on dist-test-slave-wt2j"
server_key: "cfeb6d457e0ae77e1a1d44acdaada40e"
server_key_iv: "4ba30f09f7805a811f26e62aa47e8d00"
server_key_version: "encryptionkey@0"
I20260926 16:49:31.409312 32055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "2e07ee7c976749e9afb5290d785f326c"
format_stamp: "Formatted at 2026-09-26 16:49:31 on dist-test-slave-wt2j"
server_key: "cfeb6d457e0ae77e1a1d44acdaada40e"
server_key_iv: "4ba30f09f7805a811f26e62aa47e8d00"
server_key_version: "encryptionkey@0"
I20260926 16:49:31.416604 32055 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260926 16:49:31.421674 32070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:49:31.423811 32055 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260926 16:49:31.424059 32055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "2e07ee7c976749e9afb5290d785f326c"
format_stamp: "Formatted at 2026-09-26 16:49:31 on dist-test-slave-wt2j"
server_key: "cfeb6d457e0ae77e1a1d44acdaada40e"
server_key_iv: "4ba30f09f7805a811f26e62aa47e8d00"
server_key_version: "encryptionkey@0"
I20260926 16:49:31.424317 32055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:49:31.480928 32055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:49:31.486976 32055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:49:31.487337 32055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:49:31.512229 32055 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:41281
I20260926 16:49:31.512238 32121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:41281 every 8 connection(s)
I20260926 16:49:31.514132 32055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:49:31.518463 32122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:49:31.519273 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 32055
I20260926 16:49:31.519519 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:49:31.520080 10171 external_mini_cluster.cc:1428] Setting key e5c1476f5420cd5430376e86f0878e24
I20260926 16:49:31.533785 32122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e07ee7c976749e9afb5290d785f326c: Bootstrap starting.
I20260926 16:49:31.537565 32125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:31.522452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45891 (local address 127.9.238.254:41281)
0926 16:49:31.523504 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:31.523524 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:31.523960 (+ 436us) server_negotiation.cc:407] Connection header received
0926 16:49:31.526151 (+ 2191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:31.526184 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:31.526779 (+ 595us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:31.527316 (+ 537us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:31.528027 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:31.529577 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:31.531197 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:31.532104 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:31.532333 (+ 229us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:31.534133 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:31.534144 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:31.534167 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:31.535018 (+ 851us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:31.535136 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:31.535172 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:31.535529 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:31.535902 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 16:49:31.536181 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":455,"threads_started":1}
I20260926 16:49:31.537804 32122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e07ee7c976749e9afb5290d785f326c: Neither blocks nor log segments found. Creating new log.
I20260926 16:49:31.539351 32122 log.cc:824] T 00000000000000000000000000000000 P 2e07ee7c976749e9afb5290d785f326c: Log is configured to *not* fsync() on all Append() calls
I20260926 16:49:31.543061 32122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e07ee7c976749e9afb5290d785f326c: No bootstrap required, opened a new log
I20260926 16:49:31.543251 10171 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:49:31.551754 32122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e07ee7c976749e9afb5290d785f326c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e07ee7c976749e9afb5290d785f326c" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 41281 } }
I20260926 16:49:31.552278 32122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e07ee7c976749e9afb5290d785f326c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:49:31.552400 32122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e07ee7c976749e9afb5290d785f326c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e07ee7c976749e9afb5290d785f326c, State: Initialized, Role: FOLLOWER
I20260926 16:49:31.553115 32122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e07ee7c976749e9afb5290d785f326c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e07ee7c976749e9afb5290d785f326c" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 41281 } }
I20260926 16:49:31.553409 32122 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e07ee7c976749e9afb5290d785f326c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:49:31.553556 32122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e07ee7c976749e9afb5290d785f326c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:49:31.553748 32122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e07ee7c976749e9afb5290d785f326c [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:49:31.557312 32122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e07ee7c976749e9afb5290d785f326c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e07ee7c976749e9afb5290d785f326c" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 41281 } }
I20260926 16:49:31.557710 32125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:31.548854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37089 (local address 127.9.238.254:41281)
0926 16:49:31.549172 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:31.549183 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:31.549715 (+ 532us) server_negotiation.cc:407] Connection header received
0926 16:49:31.549970 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:31.549975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:31.550067 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:31.550190 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:31.550859 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:31.551860 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:31.553936 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:31.554683 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:31.554802 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:31.556654 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:31.556661 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:31.556664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:31.556690 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:31.556785 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:31.556807 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:31.556934 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:31.557196 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 16:49:31.557264 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260926 16:49:31.558040 32122 leader_election.cc:302] T 00000000000000000000000000000000 P 2e07ee7c976749e9afb5290d785f326c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e07ee7c976749e9afb5290d785f326c; no voters:
I20260926 16:49:31.558876 32122 leader_election.cc:288] T 00000000000000000000000000000000 P 2e07ee7c976749e9afb5290d785f326c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:49:31.559051 32132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e07ee7c976749e9afb5290d785f326c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:49:31.560755 32132 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e07ee7c976749e9afb5290d785f326c [term 1 LEADER]: Becoming Leader. State: Replica: 2e07ee7c976749e9afb5290d785f326c, State: Running, Role: LEADER
I20260926 16:49:31.561465 32132 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e07ee7c976749e9afb5290d785f326c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e07ee7c976749e9afb5290d785f326c" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 41281 } }
I20260926 16:49:31.561815 32122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e07ee7c976749e9afb5290d785f326c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:49:31.564744 32134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e07ee7c976749e9afb5290d785f326c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e07ee7c976749e9afb5290d785f326c. Latest consensus state: current_term: 1 leader_uuid: "2e07ee7c976749e9afb5290d785f326c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e07ee7c976749e9afb5290d785f326c" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 41281 } } }
I20260926 16:49:31.565297 32134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e07ee7c976749e9afb5290d785f326c [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:31.565934 32133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e07ee7c976749e9afb5290d785f326c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e07ee7c976749e9afb5290d785f326c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e07ee7c976749e9afb5290d785f326c" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 41281 } } }
I20260926 16:49:31.566812 32133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e07ee7c976749e9afb5290d785f326c [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:31.570919 32138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:49:31.580338 32138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:49:31.598135 32138 catalog_manager.cc:1347] Generated new cluster ID: 1850da70412d46c1ba29e37e1456d759
I20260926 16:49:31.598348 32138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:49:31.619414 32138 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:49:31.620957 32138 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:49:31.634814 32138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e07ee7c976749e9afb5290d785f326c: Generated new TSK 0
I20260926 16:49:31.635928 32138 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:49:31.651492 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 32055
W20260926 16:49:31.653713 32117 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45891
W20260926 16:49:31.655145 10171 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 32055: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:49:31.655315 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 32055
W20260926 16:49:31.673627 32040 connection.cc:535] client connection to 127.9.238.254:41281 recv error: Network error: failed to read from TLS socket (remote: 127.9.238.254:41281): Connection reset by peer (error 104)
2026-09-26T16:49:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2118 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:49:31 dist-test-slave-wt2j krb5kdc[32153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:49:31 dist-test-slave-wt2j krb5kdc[32153](info): set up 2 sockets
Sep 26 16:49:31 dist-test-slave-wt2j krb5kdc[32153](info): commencing operation
krb5kdc: starting...
W20260926 16:49:32.821314 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:49:33 dist-test-slave-wt2j krb5kdc[32153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:49:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:49:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:49:33.164578 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:44961
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:34805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:44961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:49:33.505383 32166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:49:33.505697 32166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:49:33.505789 32166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:49:33.513058 32166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:49:33.513190 32166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:49:33.513236 32166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:49:33.513289 32166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:49:33.513340 32166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:49:33.523517 32166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:34805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:44961
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:44961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:49:33.524828 32166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:49:33.526085 32166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:49:33.539289 32171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:33.539199 32174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:33.539166 32172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:49:33.540611 32166 server_base.cc:1034] running on GCE node
I20260926 16:49:33.541507 32166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:49:33.543105 32166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:49:33.544397 32166 hybrid_clock.cc:648] HybridClock initialized: now 1790441373544308 us; error 86 us; skew 500 ppm
I20260926 16:49:33.544811 32166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:49:33 dist-test-slave-wt2j krb5kdc[32153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441373, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:49:33.548381 32166 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:49:33.550053 32166 webserver.cc:460] Webserver started at http://127.9.238.254:41487/ using document root <none> and password file <none>
I20260926 16:49:33.550757 32166 fs_manager.cc:359] Metadata directory not provided
I20260926 16:49:33.550916 32166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:49:33.551259 32166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:49:33.554317 32166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "61b84a0bb8e142c0a7e239172b836052"
format_stamp: "Formatted at 2026-09-26 16:49:33 on dist-test-slave-wt2j"
server_key: "d5fadb889a7fec3f66d654fb4d24ba4a"
server_key_iv: "418517fcbaa74007c953999bb8df2464"
server_key_version: "encryptionkey@0"
I20260926 16:49:33.555124 32166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "61b84a0bb8e142c0a7e239172b836052"
format_stamp: "Formatted at 2026-09-26 16:49:33 on dist-test-slave-wt2j"
server_key: "d5fadb889a7fec3f66d654fb4d24ba4a"
server_key_iv: "418517fcbaa74007c953999bb8df2464"
server_key_version: "encryptionkey@0"
I20260926 16:49:33.560954 32166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260926 16:49:33.565629 32181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:49:33.567770 32166 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260926 16:49:33.568022 32166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "61b84a0bb8e142c0a7e239172b836052"
format_stamp: "Formatted at 2026-09-26 16:49:33 on dist-test-slave-wt2j"
server_key: "d5fadb889a7fec3f66d654fb4d24ba4a"
server_key_iv: "418517fcbaa74007c953999bb8df2464"
server_key_version: "encryptionkey@0"
I20260926 16:49:33.568267 32166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:49:33.575309 32166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:49:33.580380 32166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:49:33.580724 32166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:49:33.602658 32166 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:44961
I20260926 16:49:33.602756 32232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:44961 every 8 connection(s)
I20260926 16:49:33.604367 32166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:49:33.606619 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 32166
I20260926 16:49:33.606835 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:49:33.607450 10171 external_mini_cluster.cc:1428] Setting key ffd0f1a2b055c6154cfc7ed1670e9060
I20260926 16:49:33.610081 32233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:49:33.616572 32236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:33.609722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59711 (local address 127.9.238.254:44961)
0926 16:49:33.610735 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:33.610751 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:33.610800 (+ 49us) server_negotiation.cc:407] Connection header received
0926 16:49:33.612634 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:33.612660 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:33.613063 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:33.613509 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:33.614718 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:33.614724 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:33.614737 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:33.614751 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:33.614830 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:33.614853 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:33.615139 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:33.615285 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 16:49:33.615487 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":478,"threads_started":1}
I20260926 16:49:33.620620 10171 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:49:33.624054 32233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61b84a0bb8e142c0a7e239172b836052: Bootstrap starting.
I20260926 16:49:33.628312 32236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:33.625415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46027 (local address 127.9.238.254:44961)
0926 16:49:33.625580 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:33.625584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:33.626128 (+ 544us) server_negotiation.cc:407] Connection header received
0926 16:49:33.626322 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:33.626327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:33.626391 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:33.626501 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:33.627531 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:33.627537 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:33.627541 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:33.627594 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:33.627646 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:33.627663 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:33.627793 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:33.627913 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 16:49:33.627975 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 16:49:33.628731 32233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61b84a0bb8e142c0a7e239172b836052: Neither blocks nor log segments found. Creating new log.
I20260926 16:49:33.630347 32233 log.cc:824] T 00000000000000000000000000000000 P 61b84a0bb8e142c0a7e239172b836052: Log is configured to *not* fsync() on all Append() calls
I20260926 16:49:33.633864 32233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61b84a0bb8e142c0a7e239172b836052: No bootstrap required, opened a new log
I20260926 16:49:33.642760 32233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61b84a0bb8e142c0a7e239172b836052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b84a0bb8e142c0a7e239172b836052" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44961 } }
I20260926 16:49:33.643328 32233 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61b84a0bb8e142c0a7e239172b836052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:49:33.643468 32233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61b84a0bb8e142c0a7e239172b836052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b84a0bb8e142c0a7e239172b836052, State: Initialized, Role: FOLLOWER
I20260926 16:49:33.644428 32233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61b84a0bb8e142c0a7e239172b836052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b84a0bb8e142c0a7e239172b836052" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44961 } }
I20260926 16:49:33.644784 32233 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61b84a0bb8e142c0a7e239172b836052 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:49:33.644972 32233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61b84a0bb8e142c0a7e239172b836052 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:49:33.645174 32233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61b84a0bb8e142c0a7e239172b836052 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:49:33.647817 32233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61b84a0bb8e142c0a7e239172b836052 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b84a0bb8e142c0a7e239172b836052" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44961 } }
I20260926 16:49:33.648458 32233 leader_election.cc:302] T 00000000000000000000000000000000 P 61b84a0bb8e142c0a7e239172b836052 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61b84a0bb8e142c0a7e239172b836052; no voters:
I20260926 16:49:33.649212 32233 leader_election.cc:288] T 00000000000000000000000000000000 P 61b84a0bb8e142c0a7e239172b836052 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:49:33.649506 32243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61b84a0bb8e142c0a7e239172b836052 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:49:33.650561 32243 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61b84a0bb8e142c0a7e239172b836052 [term 1 LEADER]: Becoming Leader. State: Replica: 61b84a0bb8e142c0a7e239172b836052, State: Running, Role: LEADER
I20260926 16:49:33.651333 32243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61b84a0bb8e142c0a7e239172b836052 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b84a0bb8e142c0a7e239172b836052" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44961 } }
I20260926 16:49:33.652133 32233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61b84a0bb8e142c0a7e239172b836052 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:49:33.654884 32245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61b84a0bb8e142c0a7e239172b836052 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61b84a0bb8e142c0a7e239172b836052. Latest consensus state: current_term: 1 leader_uuid: "61b84a0bb8e142c0a7e239172b836052" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b84a0bb8e142c0a7e239172b836052" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44961 } } }
I20260926 16:49:33.655382 32245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61b84a0bb8e142c0a7e239172b836052 [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:33.655022 32244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61b84a0bb8e142c0a7e239172b836052 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61b84a0bb8e142c0a7e239172b836052" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b84a0bb8e142c0a7e239172b836052" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 44961 } } }
I20260926 16:49:33.655596 32244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61b84a0bb8e142c0a7e239172b836052 [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:33.657992 32250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:49:33.666205 32250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:49:33.678141 32250 catalog_manager.cc:1347] Generated new cluster ID: d9bf08931e8a497581cd66344a2df2d1
I20260926 16:49:33.678288 32250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:49:33.706105 32250 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:49:33.706691 32250 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:49:33.719024 32250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61b84a0bb8e142c0a7e239172b836052: Generated new TSK 0
I20260926 16:49:33.720373 32250 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:49:33.736577 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 32166
W20260926 16:49:33.738478 32228 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59711
W20260926 16:49:33.739401 10171 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 32166: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:49:33.739656 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 32166
2026-09-26T16:49:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2087 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:49:33 dist-test-slave-wt2j krb5kdc[32264](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:49:33 dist-test-slave-wt2j krb5kdc[32264](info): set up 2 sockets
Sep 26 16:49:33 dist-test-slave-wt2j krb5kdc[32264](info): commencing operation
krb5kdc: starting...
W20260926 16:49:34.889564 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:49:35 dist-test-slave-wt2j krb5kdc[32264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:49:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:49:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:49:35.208845 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:41839
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:41839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:49:35.550437 32277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:49:35.550748 32277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:49:35.550904 32277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:49:35.558389 32277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:49:35.558524 32277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:49:35.558573 32277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:49:35.558681 32277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:49:35.558765 32277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:49:35.568944 32277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:41839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:41839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:49:35.570343 32277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:49:35.571691 32277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:49:35.585029 32285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:35.585093 32283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:35.585013 32282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:49:35.586664 32277 server_base.cc:1034] running on GCE node
I20260926 16:49:35.587743 32277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:49:35.589459 32277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:49:35.590680 32277 hybrid_clock.cc:648] HybridClock initialized: now 1790441375590618 us; error 74 us; skew 500 ppm
I20260926 16:49:35.591074 32277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:49:35 dist-test-slave-wt2j krb5kdc[32264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441375, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:49:35.599752 32277 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:49:35.601599 32277 webserver.cc:460] Webserver started at http://127.9.238.254:36071/ using document root <none> and password file <none>
I20260926 16:49:35.602339 32277 fs_manager.cc:359] Metadata directory not provided
I20260926 16:49:35.602450 32277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:49:35.602876 32277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:49:35.606083 32277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "63de7c1bb67d41ea9e2a8fe2948cdfa5"
format_stamp: "Formatted at 2026-09-26 16:49:35 on dist-test-slave-wt2j"
server_key: "65bcda9597234e52eb4b4831d5a8f17c"
server_key_iv: "86dab1c510dbe89ab8931f64dcaa2bbf"
server_key_version: "encryptionkey@0"
I20260926 16:49:35.606854 32277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "63de7c1bb67d41ea9e2a8fe2948cdfa5"
format_stamp: "Formatted at 2026-09-26 16:49:35 on dist-test-slave-wt2j"
server_key: "65bcda9597234e52eb4b4831d5a8f17c"
server_key_iv: "86dab1c510dbe89ab8931f64dcaa2bbf"
server_key_version: "encryptionkey@0"
I20260926 16:49:35.612926 32277 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260926 16:49:35.617308 32292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:49:35.619045 32277 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260926 16:49:35.619333 32277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "63de7c1bb67d41ea9e2a8fe2948cdfa5"
format_stamp: "Formatted at 2026-09-26 16:49:35 on dist-test-slave-wt2j"
server_key: "65bcda9597234e52eb4b4831d5a8f17c"
server_key_iv: "86dab1c510dbe89ab8931f64dcaa2bbf"
server_key_version: "encryptionkey@0"
I20260926 16:49:35.619603 32277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:49:35.666658 32277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:49:35.671348 32277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:49:35.671725 32277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:49:35.693753 32277 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:41839
I20260926 16:49:35.693756 32343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:41839 every 8 connection(s)
I20260926 16:49:35.695259 32277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:49:35.700474 32344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:49:35.702406 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 32277
I20260926 16:49:35.702639 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:49:35.703401 10171 external_mini_cluster.cc:1428] Setting key 4f96f0bfbd096478c161621bff82db56
I20260926 16:49:35.717788 32344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63de7c1bb67d41ea9e2a8fe2948cdfa5: Bootstrap starting.
Sep 26 16:49:35 dist-test-slave-wt2j krb5kdc[32264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:49:35.722643 32344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63de7c1bb67d41ea9e2a8fe2948cdfa5: Neither blocks nor log segments found. Creating new log.
I20260926 16:49:35.724485 32344 log.cc:824] T 00000000000000000000000000000000 P 63de7c1bb67d41ea9e2a8fe2948cdfa5: Log is configured to *not* fsync() on all Append() calls
I20260926 16:49:35.729254 32344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63de7c1bb67d41ea9e2a8fe2948cdfa5: No bootstrap required, opened a new log
I20260926 16:49:35.731429 32347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:35.706082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51903 (local address 127.9.238.254:41839)
0926 16:49:35.707203 (+ 1121us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:35.707220 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:35.707274 (+ 54us) server_negotiation.cc:407] Connection header received
0926 16:49:35.709278 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:35.709304 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:35.709796 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:35.710249 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:35.711763 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:35.713096 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:35.714372 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:35.715215 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:35.715426 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:35.721207 (+ 5781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:35.721217 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:35.721237 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:35.721287 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:35.725609 (+ 4322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:35.726843 (+ 1234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:35.726853 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:35.726863 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:35.727130 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:35.727761 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:35.727766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:35.727769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:35.728560 (+ 791us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:35.728898 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:35.729432 (+ 534us) server_negotiation.cc:299] Negotiation successful
0926 16:49:35.729971 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":460,"threads_started":1}
I20260926 16:49:35.737145 10171 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:49:35.741410 32344 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63de7c1bb67d41ea9e2a8fe2948cdfa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63de7c1bb67d41ea9e2a8fe2948cdfa5" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 41839 } }
I20260926 16:49:35.742019 32344 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63de7c1bb67d41ea9e2a8fe2948cdfa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:49:35.742172 32344 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63de7c1bb67d41ea9e2a8fe2948cdfa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63de7c1bb67d41ea9e2a8fe2948cdfa5, State: Initialized, Role: FOLLOWER
I20260926 16:49:35.742889 32344 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63de7c1bb67d41ea9e2a8fe2948cdfa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63de7c1bb67d41ea9e2a8fe2948cdfa5" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 41839 } }
I20260926 16:49:35.743184 32344 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63de7c1bb67d41ea9e2a8fe2948cdfa5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:49:35.743422 32344 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63de7c1bb67d41ea9e2a8fe2948cdfa5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:49:35.743664 32344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63de7c1bb67d41ea9e2a8fe2948cdfa5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:49:35.746416 32344 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63de7c1bb67d41ea9e2a8fe2948cdfa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63de7c1bb67d41ea9e2a8fe2948cdfa5" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 41839 } }
I20260926 16:49:35.746994 32344 leader_election.cc:302] T 00000000000000000000000000000000 P 63de7c1bb67d41ea9e2a8fe2948cdfa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63de7c1bb67d41ea9e2a8fe2948cdfa5; no voters:
I20260926 16:49:35.747830 32344 leader_election.cc:288] T 00000000000000000000000000000000 P 63de7c1bb67d41ea9e2a8fe2948cdfa5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:49:35.747949 32354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63de7c1bb67d41ea9e2a8fe2948cdfa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:49:35.749305 32354 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63de7c1bb67d41ea9e2a8fe2948cdfa5 [term 1 LEADER]: Becoming Leader. State: Replica: 63de7c1bb67d41ea9e2a8fe2948cdfa5, State: Running, Role: LEADER
I20260926 16:49:35.750048 32354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63de7c1bb67d41ea9e2a8fe2948cdfa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63de7c1bb67d41ea9e2a8fe2948cdfa5" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 41839 } }
I20260926 16:49:35.750622 32344 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63de7c1bb67d41ea9e2a8fe2948cdfa5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:49:35.753865 32355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63de7c1bb67d41ea9e2a8fe2948cdfa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63de7c1bb67d41ea9e2a8fe2948cdfa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63de7c1bb67d41ea9e2a8fe2948cdfa5" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 41839 } } }
I20260926 16:49:35.754468 32356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63de7c1bb67d41ea9e2a8fe2948cdfa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63de7c1bb67d41ea9e2a8fe2948cdfa5. Latest consensus state: current_term: 1 leader_uuid: "63de7c1bb67d41ea9e2a8fe2948cdfa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63de7c1bb67d41ea9e2a8fe2948cdfa5" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 41839 } } }
I20260926 16:49:35.755419 32355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63de7c1bb67d41ea9e2a8fe2948cdfa5 [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:35.755707 32356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63de7c1bb67d41ea9e2a8fe2948cdfa5 [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:35.767355 32347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:35.743674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43137 (local address 127.9.238.254:41839)
0926 16:49:35.743921 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:35.743927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:35.744577 (+ 650us) server_negotiation.cc:407] Connection header received
0926 16:49:35.744749 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:35.744758 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:35.744858 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:35.745007 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:35.746675 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:35.747707 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:35.749705 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:35.750258 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:35.750392 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:35.754813 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:35.754829 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:35.754835 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:35.754885 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:35.761239 (+ 6354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:35.762678 (+ 1439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:35.762687 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:35.762694 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:35.762930 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:35.763606 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:35.763614 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:35.763620 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:35.764262 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:35.764637 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:35.766346 (+ 1709us) server_negotiation.cc:299] Negotiation successful
0926 16:49:35.766744 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 16:49:35.769001 32360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:49:35.775090 32360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:49:35.786223 32360 catalog_manager.cc:1347] Generated new cluster ID: 0b8975832340438381fb58b5da876cd2
I20260926 16:49:35.786443 32360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:49:35.819422 32360 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:49:35.821615 32360 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:49:35.832145 32360 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63de7c1bb67d41ea9e2a8fe2948cdfa5: Generated new TSK 0
I20260926 16:49:35.833382 32360 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:49:35.846180 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 32277
I20260926 16:49:35.847453 32339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:49:35.946887 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 32277
2026-09-26T16:49:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2217 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:49:36 dist-test-slave-wt2j krb5kdc[32376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:49:36 dist-test-slave-wt2j krb5kdc[32376](info): set up 2 sockets
Sep 26 16:49:36 dist-test-slave-wt2j krb5kdc[32376](info): commencing operation
krb5kdc: starting...
W20260926 16:49:37.185359 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.183s user 0.002s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:49:37 dist-test-slave-wt2j krb5kdc[32376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441377, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:49:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:49:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:49:37.520254 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:45917
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:41333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:45917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:49:37.944846 32389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:49:37.945235 32389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:49:37.945411 32389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:49:37.952986 32389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:49:37.953150 32389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:49:37.953260 32389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:49:37.953374 32389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:49:37.953472 32389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:49:37.963696 32389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:41333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:45917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:45917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:49:37.965111 32389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:49:37.966506 32389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:49:37.980748 32394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:37.980827 32395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:37.981273 32397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:49:37.982221 32389 server_base.cc:1034] running on GCE node
I20260926 16:49:37.983145 32389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:49:37.984853 32389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:49:37.986086 32389 hybrid_clock.cc:648] HybridClock initialized: now 1790441377986007 us; error 78 us; skew 500 ppm
I20260926 16:49:37.986501 32389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:49:37 dist-test-slave-wt2j krb5kdc[32376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441377, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:49:37.989977 32389 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:49:37.991894 32389 webserver.cc:460] Webserver started at http://127.9.238.254:32841/ using document root <none> and password file <none>
I20260926 16:49:37.992674 32389 fs_manager.cc:359] Metadata directory not provided
I20260926 16:49:37.992784 32389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:49:37.993144 32389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:49:37.996373 32389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "fd80c0f1c9754bcfa9a8bb9861a8a7ff"
format_stamp: "Formatted at 2026-09-26 16:49:37 on dist-test-slave-wt2j"
server_key: "e7180d03bae74ba7f68f73035809e160"
server_key_iv: "e5e521500e8d07b57bc72243cab60122"
server_key_version: "encryptionkey@0"
I20260926 16:49:37.997162 32389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "fd80c0f1c9754bcfa9a8bb9861a8a7ff"
format_stamp: "Formatted at 2026-09-26 16:49:37 on dist-test-slave-wt2j"
server_key: "e7180d03bae74ba7f68f73035809e160"
server_key_iv: "e5e521500e8d07b57bc72243cab60122"
server_key_version: "encryptionkey@0"
I20260926 16:49:38.004104 32389 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260926 16:49:38.009312 32404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:49:38.011158 32389 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260926 16:49:38.011427 32389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "fd80c0f1c9754bcfa9a8bb9861a8a7ff"
format_stamp: "Formatted at 2026-09-26 16:49:37 on dist-test-slave-wt2j"
server_key: "e7180d03bae74ba7f68f73035809e160"
server_key_iv: "e5e521500e8d07b57bc72243cab60122"
server_key_version: "encryptionkey@0"
I20260926 16:49:38.011746 32389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:49:38.030450 32389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:49:38.035615 32389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:49:38.035928 32389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:49:38.057981 32389 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:45917
I20260926 16:49:38.058049 32455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:45917 every 8 connection(s)
I20260926 16:49:38.059659 32389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:49:38.065001 32456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:49:38.065179 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 32389
I20260926 16:49:38.065388 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:49:38.065989 10171 external_mini_cluster.cc:1428] Setting key cd32272990cd618ddca559297223cb4a
I20260926 16:49:38.082325 32456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd80c0f1c9754bcfa9a8bb9861a8a7ff: Bootstrap starting.
I20260926 16:49:38.083654 32459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:38.068474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45977 (local address 127.9.238.254:45917)
0926 16:49:38.069594 (+ 1120us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:38.069607 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:38.069659 (+ 52us) server_negotiation.cc:407] Connection header received
0926 16:49:38.072778 (+ 3119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:38.072812 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:38.073340 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:38.073758 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:38.074672 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:38.076283 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:38.078376 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:38.079306 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:38.079461 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:38.081049 (+ 1588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:38.081058 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:38.081074 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:38.081696 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:38.081792 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:38.081825 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:38.081929 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:38.082215 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 16:49:38.082508 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":845,"thread_start_us":542,"threads_started":1}
I20260926 16:49:38.086829 32456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd80c0f1c9754bcfa9a8bb9861a8a7ff: Neither blocks nor log segments found. Creating new log.
I20260926 16:49:38.088433 10171 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:49:38.088637 32456 log.cc:824] T 00000000000000000000000000000000 P fd80c0f1c9754bcfa9a8bb9861a8a7ff: Log is configured to *not* fsync() on all Append() calls
I20260926 16:49:38.093212 32456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd80c0f1c9754bcfa9a8bb9861a8a7ff: No bootstrap required, opened a new log
I20260926 16:49:38.102617 32459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:38.094721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51993 (local address 127.9.238.254:45917)
0926 16:49:38.094973 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:38.094980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:38.095529 (+ 549us) server_negotiation.cc:407] Connection header received
0926 16:49:38.095805 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:38.095813 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:38.095927 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:38.096109 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:38.096658 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:38.097909 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:38.099375 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:38.100208 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:38.100384 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:38.101680 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:38.101687 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:38.101692 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:38.101721 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:38.101780 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:38.101801 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:38.101904 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:38.102051 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 16:49:38.102194 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 16:49:38.105846 32456 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd80c0f1c9754bcfa9a8bb9861a8a7ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd80c0f1c9754bcfa9a8bb9861a8a7ff" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 45917 } }
I20260926 16:49:38.106431 32456 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd80c0f1c9754bcfa9a8bb9861a8a7ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:49:38.106520 32456 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd80c0f1c9754bcfa9a8bb9861a8a7ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd80c0f1c9754bcfa9a8bb9861a8a7ff, State: Initialized, Role: FOLLOWER
I20260926 16:49:38.107381 32456 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd80c0f1c9754bcfa9a8bb9861a8a7ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd80c0f1c9754bcfa9a8bb9861a8a7ff" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 45917 } }
I20260926 16:49:38.107852 32456 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd80c0f1c9754bcfa9a8bb9861a8a7ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:49:38.108047 32456 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd80c0f1c9754bcfa9a8bb9861a8a7ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:49:38.108304 32456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd80c0f1c9754bcfa9a8bb9861a8a7ff [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:49:38.111048 32456 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd80c0f1c9754bcfa9a8bb9861a8a7ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd80c0f1c9754bcfa9a8bb9861a8a7ff" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 45917 } }
I20260926 16:49:38.111939 32456 leader_election.cc:302] T 00000000000000000000000000000000 P fd80c0f1c9754bcfa9a8bb9861a8a7ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd80c0f1c9754bcfa9a8bb9861a8a7ff; no voters:
I20260926 16:49:38.112903 32456 leader_election.cc:288] T 00000000000000000000000000000000 P fd80c0f1c9754bcfa9a8bb9861a8a7ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:49:38.113133 32466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd80c0f1c9754bcfa9a8bb9861a8a7ff [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:49:38.114655 32466 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd80c0f1c9754bcfa9a8bb9861a8a7ff [term 1 LEADER]: Becoming Leader. State: Replica: fd80c0f1c9754bcfa9a8bb9861a8a7ff, State: Running, Role: LEADER
I20260926 16:49:38.115527 32466 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd80c0f1c9754bcfa9a8bb9861a8a7ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd80c0f1c9754bcfa9a8bb9861a8a7ff" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 45917 } }
I20260926 16:49:38.116426 32456 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd80c0f1c9754bcfa9a8bb9861a8a7ff [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:49:38.119340 32468 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd80c0f1c9754bcfa9a8bb9861a8a7ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd80c0f1c9754bcfa9a8bb9861a8a7ff. Latest consensus state: current_term: 1 leader_uuid: "fd80c0f1c9754bcfa9a8bb9861a8a7ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd80c0f1c9754bcfa9a8bb9861a8a7ff" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 45917 } } }
I20260926 16:49:38.119289 32467 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd80c0f1c9754bcfa9a8bb9861a8a7ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd80c0f1c9754bcfa9a8bb9861a8a7ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd80c0f1c9754bcfa9a8bb9861a8a7ff" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 45917 } } }
I20260926 16:49:38.120049 32468 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd80c0f1c9754bcfa9a8bb9861a8a7ff [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:38.120049 32467 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd80c0f1c9754bcfa9a8bb9861a8a7ff [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:38.122460 32473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:49:38.129134 32473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:49:38.141109 32473 catalog_manager.cc:1347] Generated new cluster ID: f928464934654a7ba3d87804ec00d326
I20260926 16:49:38.141361 32473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:49:38.169140 32473 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:49:38.171468 32473 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:49:38.190930 32473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd80c0f1c9754bcfa9a8bb9861a8a7ff: Generated new TSK 0
I20260926 16:49:38.192279 32473 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:49:38.204936 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 32389
W20260926 16:49:38.206327 32451 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45977
W20260926 16:49:38.207273 10171 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 32389: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:49:38.207404 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 32389
2026-09-26T16:49:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2250 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:49:38 dist-test-slave-wt2j krb5kdc[32487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:49:38 dist-test-slave-wt2j krb5kdc[32487](info): set up 2 sockets
Sep 26 16:49:38 dist-test-slave-wt2j krb5kdc[32487](info): commencing operation
krb5kdc: starting...
W20260926 16:49:39.362012 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:49:39 dist-test-slave-wt2j krb5kdc[32487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:49:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:49:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:49:39.688133 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:46861
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:46145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:46861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:49:40.100876 32500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:49:40.101217 32500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:49:40.101316 32500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:49:40.109057 32500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:49:40.109206 32500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:49:40.109254 32500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:49:40.109310 32500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:49:40.109354 32500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:49:40.119746 32500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:46145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:46861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:46861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:49:40.121173 32500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:49:40.122594 32500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:49:40.137856 32506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:40.138113 32508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:40.138332 32505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:49:40.138914 32500 server_base.cc:1034] running on GCE node
I20260926 16:49:40.140170 32500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:49:40.142333 32500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:49:40.143683 32500 hybrid_clock.cc:648] HybridClock initialized: now 1790441380143595 us; error 53 us; skew 500 ppm
I20260926 16:49:40.144169 32500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:49:40 dist-test-slave-wt2j krb5kdc[32487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441380, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:49:40.148559 32500 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:49:40.150880 32500 webserver.cc:460] Webserver started at http://127.9.238.254:35777/ using document root <none> and password file <none>
I20260926 16:49:40.151947 32500 fs_manager.cc:359] Metadata directory not provided
I20260926 16:49:40.152110 32500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:49:40.152518 32500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:49:40.156991 32500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "fbc83b61f9904f5083e8f2a9a327e177"
format_stamp: "Formatted at 2026-09-26 16:49:40 on dist-test-slave-wt2j"
server_key: "0e203467e7bdaf03547898d1d57862d7"
server_key_iv: "68d25ac48597da7b2a8646a91668397f"
server_key_version: "encryptionkey@0"
I20260926 16:49:40.158118 32500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "fbc83b61f9904f5083e8f2a9a327e177"
format_stamp: "Formatted at 2026-09-26 16:49:40 on dist-test-slave-wt2j"
server_key: "0e203467e7bdaf03547898d1d57862d7"
server_key_iv: "68d25ac48597da7b2a8646a91668397f"
server_key_version: "encryptionkey@0"
I20260926 16:49:40.165802 32500 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260926 16:49:40.170876 32515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:49:40.174662 32500 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260926 16:49:40.176141 32500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "fbc83b61f9904f5083e8f2a9a327e177"
format_stamp: "Formatted at 2026-09-26 16:49:40 on dist-test-slave-wt2j"
server_key: "0e203467e7bdaf03547898d1d57862d7"
server_key_iv: "68d25ac48597da7b2a8646a91668397f"
server_key_version: "encryptionkey@0"
I20260926 16:49:40.176733 32500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:49:40.185460 32500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:49:40.191987 32500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:49:40.192438 32500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:49:40.217917 32500 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:46861
I20260926 16:49:40.217952 32566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:46861 every 8 connection(s)
I20260926 16:49:40.220009 32500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:49:40.224570 32567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:49:40.225140 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 32500
I20260926 16:49:40.225324 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:49:40.225898 10171 external_mini_cluster.cc:1428] Setting key 240a1e4dcd9785297e52b2fbff5248fd
I20260926 16:49:40.236701 32570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:40.228391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41919 (local address 127.9.238.254:46861)
0926 16:49:40.229521 (+ 1130us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:40.229536 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:40.229594 (+ 58us) server_negotiation.cc:407] Connection header received
0926 16:49:40.231814 (+ 2220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:40.231849 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:40.232398 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:40.232899 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:40.234293 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:40.234302 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:40.234319 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:40.234340 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:40.234443 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:40.234474 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:40.234737 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:40.234869 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 16:49:40.235258 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":827,"thread_start_us":520,"threads_started":1}
I20260926 16:49:40.239998 32567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbc83b61f9904f5083e8f2a9a327e177: Bootstrap starting.
I20260926 16:49:40.241870 10171 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:49:40.244128 32567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbc83b61f9904f5083e8f2a9a327e177: Neither blocks nor log segments found. Creating new log.
I20260926 16:49:40.245616 32567 log.cc:824] T 00000000000000000000000000000000 P fbc83b61f9904f5083e8f2a9a327e177: Log is configured to *not* fsync() on all Append() calls
I20260926 16:49:40.249760 32567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbc83b61f9904f5083e8f2a9a327e177: No bootstrap required, opened a new log
I20260926 16:49:40.251199 32570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:40.247790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58447 (local address 127.9.238.254:46861)
0926 16:49:40.248052 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:40.248059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:40.248757 (+ 698us) server_negotiation.cc:407] Connection header received
0926 16:49:40.249018 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:40.249025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:40.249094 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:40.249269 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:40.250458 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:40.250464 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:40.250469 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 16:49:40.250491 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:40.250548 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 16:49:40.250575 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:40.250720 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:40.250829 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 16:49:40.250894 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 16:49:40.258816 32567 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbc83b61f9904f5083e8f2a9a327e177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc83b61f9904f5083e8f2a9a327e177" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46861 } }
I20260926 16:49:40.259294 32567 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbc83b61f9904f5083e8f2a9a327e177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 16:49:40.259400 32567 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbc83b61f9904f5083e8f2a9a327e177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc83b61f9904f5083e8f2a9a327e177, State: Initialized, Role: FOLLOWER
I20260926 16:49:40.260145 32567 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbc83b61f9904f5083e8f2a9a327e177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc83b61f9904f5083e8f2a9a327e177" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46861 } }
I20260926 16:49:40.260385 32567 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbc83b61f9904f5083e8f2a9a327e177 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 16:49:40.260560 32567 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbc83b61f9904f5083e8f2a9a327e177 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 16:49:40.260767 32567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbc83b61f9904f5083e8f2a9a327e177 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:49:40.263064 32567 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbc83b61f9904f5083e8f2a9a327e177 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc83b61f9904f5083e8f2a9a327e177" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46861 } }
I20260926 16:49:40.263681 32567 leader_election.cc:302] T 00000000000000000000000000000000 P fbc83b61f9904f5083e8f2a9a327e177 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbc83b61f9904f5083e8f2a9a327e177; no voters:
I20260926 16:49:40.264389 32567 leader_election.cc:288] T 00000000000000000000000000000000 P fbc83b61f9904f5083e8f2a9a327e177 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 16:49:40.264642 32577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbc83b61f9904f5083e8f2a9a327e177 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:49:40.265709 32577 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbc83b61f9904f5083e8f2a9a327e177 [term 1 LEADER]: Becoming Leader. State: Replica: fbc83b61f9904f5083e8f2a9a327e177, State: Running, Role: LEADER
I20260926 16:49:40.266484 32577 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbc83b61f9904f5083e8f2a9a327e177 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc83b61f9904f5083e8f2a9a327e177" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46861 } }
I20260926 16:49:40.266845 32567 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbc83b61f9904f5083e8f2a9a327e177 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:49:40.271257 32579 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbc83b61f9904f5083e8f2a9a327e177 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbc83b61f9904f5083e8f2a9a327e177. Latest consensus state: current_term: 1 leader_uuid: "fbc83b61f9904f5083e8f2a9a327e177" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc83b61f9904f5083e8f2a9a327e177" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46861 } } }
I20260926 16:49:40.272068 32579 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbc83b61f9904f5083e8f2a9a327e177 [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:40.271767 32578 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbc83b61f9904f5083e8f2a9a327e177 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbc83b61f9904f5083e8f2a9a327e177" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbc83b61f9904f5083e8f2a9a327e177" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 46861 } } }
I20260926 16:49:40.272214 32578 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbc83b61f9904f5083e8f2a9a327e177 [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:40.277588 32587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:49:40.284586 32587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:49:40.296993 32587 catalog_manager.cc:1347] Generated new cluster ID: 18d54150f7a14c26a152b081b8bb1054
I20260926 16:49:40.297174 32587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:49:40.343834 32587 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:49:40.344550 32587 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:49:40.360342 32587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbc83b61f9904f5083e8f2a9a327e177: Generated new TSK 0
I20260926 16:49:40.361508 32587 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:49:40.385236 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 32500
W20260926 16:49:40.386674 32562 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41919
W20260926 16:49:40.387672 10171 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 32500: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260926 16:49:40.387862 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 32500
2026-09-26T16:49:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2177 ms)
[----------] 18 tests from AuthTokenIssuingTest (39101 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:49:40 dist-test-slave-wt2j krb5kdc[32598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:49:40 dist-test-slave-wt2j krb5kdc[32598](info): set up 2 sockets
Sep 26 16:49:40 dist-test-slave-wt2j krb5kdc[32598](info): commencing operation
krb5kdc: starting...
W20260926 16:49:41.537464 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.004s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:49:41 dist-test-slave-wt2j krb5kdc[32598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:49:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:49:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:49:41.868566 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:40099
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:40099,127.9.238.253:34035,127.9.238.252:43535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:49:42.263312 32611 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 16:49:42.263737 32611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:49:42.263821 32611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:49:42.263911 32611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:49:42.271466 32611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:49:42.271667 32611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:49:42.271742 32611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:49:42.271818 32611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:49:42.271888 32611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:49:42.282084 32611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:40099,127.9.238.253:34035,127.9.238.252:43535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:40099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:49:42.283442 32611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:49:42.284884 32611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:49:42.298676 32616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:42.298527 32617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:42.298833 32619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:49:42.299810 32611 server_base.cc:1034] running on GCE node
I20260926 16:49:42.300627 32611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:49:42.302227 32611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:49:42.303478 32611 hybrid_clock.cc:648] HybridClock initialized: now 1790441382303414 us; error 33 us; skew 500 ppm
I20260926 16:49:42.303913 32611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:49:42 dist-test-slave-wt2j krb5kdc[32598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441382, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:49:42.315387 32611 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:49:42.317484 32611 webserver.cc:460] Webserver started at http://127.9.238.254:37827/ using document root <none> and password file <none>
I20260926 16:49:42.318284 32611 fs_manager.cc:359] Metadata directory not provided
I20260926 16:49:42.318387 32611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:49:42.318770 32611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:49:42.322093 32611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "cbaf040a5d974d408fafd27446e03b3c"
format_stamp: "Formatted at 2026-09-26 16:49:42 on dist-test-slave-wt2j"
server_key: "ce6c1ed991f327ce6d58dfb55b244bcd"
server_key_iv: "276fd711ffac89e6f90d2ca304d1a57a"
server_key_version: "encryptionkey@0"
I20260926 16:49:42.322968 32611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "cbaf040a5d974d408fafd27446e03b3c"
format_stamp: "Formatted at 2026-09-26 16:49:42 on dist-test-slave-wt2j"
server_key: "ce6c1ed991f327ce6d58dfb55b244bcd"
server_key_iv: "276fd711ffac89e6f90d2ca304d1a57a"
server_key_version: "encryptionkey@0"
I20260926 16:49:42.329237 32611 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.006s
I20260926 16:49:42.333590 32626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:49:42.335348 32611 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 16:49:42.335637 32611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "cbaf040a5d974d408fafd27446e03b3c"
format_stamp: "Formatted at 2026-09-26 16:49:42 on dist-test-slave-wt2j"
server_key: "ce6c1ed991f327ce6d58dfb55b244bcd"
server_key_iv: "276fd711ffac89e6f90d2ca304d1a57a"
server_key_version: "encryptionkey@0"
I20260926 16:49:42.335932 32611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:49:42.367656 32611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:49:42.372169 32611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:49:42.372464 32611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:49:42.394778 32611 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:40099
I20260926 16:49:42.394886 32677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:40099 every 8 connection(s)
I20260926 16:49:42.396399 32611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:49:42.401463 32678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:49:42.404808 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 32611
I20260926 16:49:42.405089 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:49:42.405978 10171 external_mini_cluster.cc:1428] Setting key e44634f3bbd90de44772f59f710e61e7
I20260926 16:49:42.413466 32678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40099 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:49:42 dist-test-slave-wt2j krb5kdc[32598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441382, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.238.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.253@KRBTEST.COM" created.
I20260926 16:49:42.446638 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:42.418864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40099 (local address 127.0.0.1:47220)
0926 16:49:42.420114 (+ 1250us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:42.420184 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 16:49:42.423065 (+ 2881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:49:42.424389 (+ 1324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:49:42.424396 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:49:42.425170 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:49:42.425626 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:42.425647 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:42.428201 (+ 2554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:42.428206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:42.429551 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:42.429565 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:42.431175 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:42.431180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:42.431402 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:42.432400 (+ 998us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:49:42.432433 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:49:42.435404 (+ 2971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:49:42.440870 (+ 5466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:42.440886 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:42.440906 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:42.441838 (+ 932us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:42.442446 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:42.442450 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:42.442453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:42.442724 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:42.443998 (+ 1274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:49:42.444011 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:49:42.444378 (+ 367us) client_negotiation.cc:769] Sending connection context
0926 16:49:42.444664 (+ 286us) client_negotiation.cc:240] Negotiation successful
0926 16:49:42.445225 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"thread_start_us":444,"threads_started":1}
I20260926 16:49:42.446758 32681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:42.418864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47220 (local address 127.9.238.254:40099)
0926 16:49:42.421167 (+ 2303us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:42.421177 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:42.421226 (+ 49us) server_negotiation.cc:407] Connection header received
0926 16:49:42.423641 (+ 2415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:42.423655 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:42.424024 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:42.424310 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:42.425910 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:42.427981 (+ 2071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:42.429777 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:42.430848 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:42.431115 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:42.435836 (+ 4721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:42.435852 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:42.435875 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:42.435935 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:42.440622 (+ 4687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:42.442020 (+ 1398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:42.442030 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:42.442032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:42.442257 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:42.442888 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:42.442893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:42.442896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:42.443802 (+ 906us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:42.444010 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:42.444890 (+ 880us) server_negotiation.cc:299] Negotiation successful
0926 16:49:42.445490 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":167,"server-negotiator.queue_time_us":2059,"spinlock_wait_cycles":34688,"thread_start_us":1751,"threads_started":1}
I20260926 16:49:42.450636 32678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.253" port: 34035 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:49:42.452383 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:42.451648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:34035 (local address 127.0.0.1:41080)
0926 16:49:42.451983 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:42.452141 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
W20260926 16:49:42.452785 32630 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.238.253:34035: connect: Connection refused (error 111)
I20260926 16:49:42.455489 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:42.454860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:34035 (local address 127.0.0.1:41086)
0926 16:49:42.455178 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:42.455306 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260926 16:49:42.455891 32678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.253:34035: Network error: Client connection negotiation failed: client connection to 127.9.238.253:34035: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.253.keytab.
Entry for principal kudu/127.9.238.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.253.keytab.
I20260926 16:49:42.484697 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.253:34035
--webserver_interface=127.9.238.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.253.keytab
--principal=kudu/127.9.238.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:40099,127.9.238.253:34035,127.9.238.252:43535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:49:42.505339 32678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.253" port: 34035 } attempt: 1
I20260926 16:49:42.506711 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:42.506175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:34035 (local address 127.0.0.1:41090)
0926 16:49:42.506391 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:42.506519 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 16:49:42.508131 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:42.507756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:34035 (local address 127.0.0.1:41106)
0926 16:49:42.507918 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:42.507995 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 16:49:42.508422 32678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.253:34035: Network error: Client connection negotiation failed: client connection to 127.9.238.253:34035: connect: Connection refused (error 111)
I20260926 16:49:42.576814 32678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.253" port: 34035 } attempt: 2
I20260926 16:49:42.578570 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:42.577831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:34035 (local address 127.0.0.1:41114)
0926 16:49:42.578215 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:42.578396 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260926 16:49:42.579888 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:42.579437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:34035 (local address 127.0.0.1:41120)
0926 16:49:42.579648 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:42.579737 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 16:49:42.580083 32678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.253:34035: Network error: Client connection negotiation failed: client connection to 127.9.238.253:34035: connect: Connection refused (error 111)
I20260926 16:49:42.671463 32678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.253" port: 34035 } attempt: 3
I20260926 16:49:42.673110 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:42.672473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:34035 (local address 127.0.0.1:41134)
0926 16:49:42.672798 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:42.672937 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260926 16:49:42.674559 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:42.674059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:34035 (local address 127.0.0.1:41138)
0926 16:49:42.674320 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:42.674399 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260926 16:49:42.674763 32678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.253:34035: Network error: Client connection negotiation failed: client connection to 127.9.238.253:34035: connect: Connection refused (error 111)
I20260926 16:49:42.818077 32678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.253" port: 34035 } attempt: 4
I20260926 16:49:42.819907 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:42.819059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:34035 (local address 127.0.0.1:41152)
0926 16:49:42.819509 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:42.819709 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229}
I20260926 16:49:42.821120 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:42.820757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:34035 (local address 127.0.0.1:41160)
0926 16:49:42.820916 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:42.821006 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:34035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 16:49:42.821357 32678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.253:34035: Network error: Client connection negotiation failed: client connection to 127.9.238.253:34035: connect: Connection refused (error 111)
W20260926 16:49:42.844687 32685 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 16:49:42.845093 32685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:49:42.845224 32685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:49:42.845367 32685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:49:42.852871 32685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:49:42.853024 32685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:49:42.853152 32685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:49:42.853240 32685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:49:42.853333 32685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:49:42.863922 32685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:40099,127.9.238.253:34035,127.9.238.252:43535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.253.keytab
--principal=kudu/127.9.238.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.253:34035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.238.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:49:42.865485 32685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:49:42.866808 32685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:49:42.880566 32693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:42.880800 32691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:42.880620 32690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:49:42.881819 32685 server_base.cc:1034] running on GCE node
I20260926 16:49:42.882911 32685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:49:42.884905 32685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:49:42.886236 32685 hybrid_clock.cc:648] HybridClock initialized: now 1790441382886142 us; error 59 us; skew 500 ppm
I20260926 16:49:42.886773 32685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:49:42 dist-test-slave-wt2j krb5kdc[32598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441382, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:49:42.890856 32685 init.cc:377] Logged in from keytab as kudu/127.9.238.253@KRBTEST.COM (short username kudu)
I20260926 16:49:42.892899 32685 webserver.cc:460] Webserver started at http://127.9.238.253:40371/ using document root <none> and password file <none>
I20260926 16:49:42.893736 32685 fs_manager.cc:359] Metadata directory not provided
I20260926 16:49:42.893836 32685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:49:42.894177 32685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:49:42.897925 32685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-1/data/instance:
uuid: "deb8a9fdd2e741c780e3c1d259870022"
format_stamp: "Formatted at 2026-09-26 16:49:42 on dist-test-slave-wt2j"
server_key: "275a955c5f320ff44b9c7236c35d0941"
server_key_iv: "3cb8c4389fb8795a2668e3550f6cfb60"
server_key_version: "encryptionkey@0"
I20260926 16:49:42.898828 32685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-1/wal/instance:
uuid: "deb8a9fdd2e741c780e3c1d259870022"
format_stamp: "Formatted at 2026-09-26 16:49:42 on dist-test-slave-wt2j"
server_key: "275a955c5f320ff44b9c7236c35d0941"
server_key_iv: "3cb8c4389fb8795a2668e3550f6cfb60"
server_key_version: "encryptionkey@0"
I20260926 16:49:42.905509 32685 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260926 16:49:42.910298 32700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:49:42.912439 32685 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:49:42.912719 32685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-1/wal
uuid: "deb8a9fdd2e741c780e3c1d259870022"
format_stamp: "Formatted at 2026-09-26 16:49:42 on dist-test-slave-wt2j"
server_key: "275a955c5f320ff44b9c7236c35d0941"
server_key_iv: "3cb8c4389fb8795a2668e3550f6cfb60"
server_key_version: "encryptionkey@0"
I20260926 16:49:42.913007 32685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:49:42.943435 32685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:49:42.949460 32685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:49:42.949851 32685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:49:42.974648 32685 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.253:34035
I20260926 16:49:42.974659 32751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.253:34035 every 8 connection(s)
I20260926 16:49:42.976568 32685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-1/data/info.pb
I20260926 16:49:42.977622 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 32685
I20260926 16:49:42.977833 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-1/wal/instance
I20260926 16:49:42.978354 10171 external_mini_cluster.cc:1428] Setting key 0d70bf76751825de61b6581ce977236b
I20260926 16:49:42.982084 32752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:49:42.993600 32752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40099 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.9.238.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.252@KRBTEST.COM" created.
Sep 26 16:49:43 dist-test-slave-wt2j krb5kdc[32598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441382, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.253@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:49:43.026115 32755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:42.998235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47230 (local address 127.9.238.254:40099)
0926 16:49:42.999019 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:42.999030 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:43.000330 (+ 1300us) server_negotiation.cc:407] Connection header received
0926 16:49:43.002541 (+ 2211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:43.002552 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:43.002689 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:43.002944 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:43.005052 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.006480 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.008542 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.009383 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.009583 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:43.017485 (+ 7902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:43.017493 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:43.017497 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:43.017567 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:43.021153 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:43.022501 (+ 1348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:43.022507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:43.022510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:43.022670 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:43.023321 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:43.023326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:43.023329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:43.023838 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:43.024001 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:43.025163 (+ 1162us) server_negotiation.cc:299] Negotiation successful
0926 16:49:43.025566 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":374,"threads_started":1}
I20260926 16:49:43.026743 32756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:42.999131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40099 (local address 127.0.0.1:47230)
0926 16:49:43.000092 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:43.000145 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 16:49:43.002205 (+ 2060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:49:43.003270 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:49:43.003279 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:49:43.004083 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:49:43.004738 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:43.004758 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.006759 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.006765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:43.008312 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:43.008331 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.009709 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.009715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:43.009971 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:43.011015 (+ 1044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:49:43.011051 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:49:43.016983 (+ 5932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:49:43.021403 (+ 4420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:43.021418 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:43.021442 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:43.022294 (+ 852us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:43.022808 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:43.022816 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:43.022822 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:43.023110 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:43.023999 (+ 889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:49:43.024017 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:49:43.024602 (+ 585us) client_negotiation.cc:769] Sending connection context
0926 16:49:43.024962 (+ 360us) client_negotiation.cc:240] Negotiation successful
0926 16:49:43.025513 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":419,"threads_started":1}
I20260926 16:49:43.029258 32752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.253" port: 34035 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:49:43 dist-test-slave-wt2j krb5kdc[32598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441382, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.253@KRBTEST.COM for kudu/127.9.238.253@KRBTEST.COM
I20260926 16:49:43.049271 32756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.030464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:34035 (local address 127.0.0.1:41174)
0926 16:49:43.030657 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:43.030679 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:49:43.030797 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:49:43.031910 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:49:43.031914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:49:43.032432 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:49:43.032585 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:43.032595 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.034044 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.034052 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:43.034898 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:43.034941 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.036490 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.036497 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:43.036694 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:43.037511 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:49:43.037531 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:49:43.040505 (+ 2974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:49:43.045789 (+ 5284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:43.045795 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:43.045798 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:43.046317 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:43.047012 (+ 695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:43.047017 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:43.047019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:43.047324 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:43.048158 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:49:43.048162 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:49:43.048353 (+ 191us) client_negotiation.cc:769] Sending connection context
0926 16:49:43.048500 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 16:49:43.048914 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 16:49:43.049860 32758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.030615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41174 (local address 127.9.238.253:34035)
0926 16:49:43.031304 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:43.031310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:43.031336 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:49:43.031429 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:43.031438 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:43.031757 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:43.031926 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:43.032766 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.033890 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.035396 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.036235 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.036441 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:43.040781 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:43.040797 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:43.040817 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:43.040855 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:43.045519 (+ 4664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:43.046548 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:43.046561 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:43.046568 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:43.046787 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:43.047494 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:43.047502 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:43.047508 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:43.047990 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:43.048180 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:43.048743 (+ 563us) server_negotiation.cc:299] Negotiation successful
0926 16:49:43.049160 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":529,"thread_start_us":357,"threads_started":1}
I20260926 16:49:43.053362 32752 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:49:43.054989 32756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.054246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:43535 (local address 127.0.0.1:59110)
0926 16:49:43.054593 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:43.054808 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:43535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260926 16:49:43.055420 32704 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.238.252:43535: connect: Connection refused (error 111)
I20260926 16:49:43.057703 32756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.057195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:43535 (local address 127.0.0.1:59118)
0926 16:49:43.057456 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:43.057570 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:43535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260926 16:49:43.058033 32752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:43535: Network error: Client connection negotiation failed: client connection to 127.9.238.252:43535: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.252.keytab.
Entry for principal kudu/127.9.238.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.252.keytab.
I20260926 16:49:43.060482 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.252:43535
--webserver_interface=127.9.238.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.252.keytab
--principal=kudu/127.9.238.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:40099,127.9.238.253:34035,127.9.238.252:43535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:49:43.107417 32752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } attempt: 1
I20260926 16:49:43.109143 32756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.108485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:43535 (local address 127.0.0.1:59132)
0926 16:49:43.108778 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:43.108932 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:43535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260926 16:49:43.110561 32756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.109995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:43535 (local address 127.0.0.1:59140)
0926 16:49:43.110253 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:43.110380 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:43535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260926 16:49:43.110885 32752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:43535: Network error: Client connection negotiation failed: client connection to 127.9.238.252:43535: connect: Connection refused (error 111)
I20260926 16:49:43.120659 32678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.253" port: 34035 } attempt: 5
Sep 26 16:49:43 dist-test-slave-wt2j krb5kdc[32598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441382, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for kudu/127.9.238.253@KRBTEST.COM
I20260926 16:49:43.138669 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.121735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:34035 (local address 127.0.0.1:41178)
0926 16:49:43.122051 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:43.122074 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:49:43.122227 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:49:43.122697 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:49:43.122701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:49:43.123199 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:49:43.123427 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:43.123437 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.125006 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.125012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:43.126051 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:43.126068 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.127284 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.127291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:43.127513 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:43.128505 (+ 992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:49:43.128530 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:49:43.131289 (+ 2759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:49:43.135662 (+ 4373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:43.135668 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:43.135672 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:43.136248 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:43.136765 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:43.136769 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:43.136771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:43.136955 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:43.137640 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:49:43.137644 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:49:43.137865 (+ 221us) client_negotiation.cc:769] Sending connection context
0926 16:49:43.138058 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 16:49:43.138364 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 16:49:43.138888 32758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.121906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41178 (local address 127.9.238.253:34035)
0926 16:49:43.122186 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:43.122193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:43.122224 (+ 31us) server_negotiation.cc:407] Connection header received
0926 16:49:43.122386 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:43.122392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:43.122485 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:43.122623 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:43.123633 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.124776 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.126234 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.126945 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.127107 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:43.131611 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:43.131619 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:43.131623 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:43.131672 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:43.135357 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:43.136429 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:43.136436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:43.136440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:43.136616 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:43.137087 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:43.137092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:43.137095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:43.137484 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:43.137655 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:43.138214 (+ 559us) server_negotiation.cc:299] Negotiation successful
0926 16:49:43.138456 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260926 16:49:43.140400 32678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:49:43.141886 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.141293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:43535 (local address 127.0.0.1:59148)
0926 16:49:43.141534 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:43.141644 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:43535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260926 16:49:43.143033 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.142647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:43535 (local address 127.0.0.1:59162)
0926 16:49:43.142816 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:43.142918 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:43535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 16:49:43.143391 32678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:43535: Network error: Client connection negotiation failed: client connection to 127.9.238.252:43535: connect: Connection refused (error 111)
I20260926 16:49:43.179170 32752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } attempt: 2
I20260926 16:49:43.180843 32756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.180195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:43535 (local address 127.0.0.1:59168)
0926 16:49:43.180470 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:43.180631 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:43535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 16:49:43.182117 32756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.181602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:43535 (local address 127.0.0.1:59174)
0926 16:49:43.181835 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:43.181945 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:43535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 16:49:43.182446 32752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:43535: Network error: Client connection negotiation failed: client connection to 127.9.238.252:43535: connect: Connection refused (error 111)
I20260926 16:49:43.194752 32678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } attempt: 1
I20260926 16:49:43.196681 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.195867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:43535 (local address 127.0.0.1:59182)
0926 16:49:43.196206 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:43.196375 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:43535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260926 16:49:43.198040 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.197571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:43535 (local address 127.0.0.1:59186)
0926 16:49:43.197786 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:43.197907 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:43535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260926 16:49:43.198328 32678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:43535: Network error: Client connection negotiation failed: client connection to 127.9.238.252:43535: connect: Connection refused (error 111)
I20260926 16:49:43.266642 32678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } attempt: 2
I20260926 16:49:43.268498 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.267733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:43535 (local address 127.0.0.1:59198)
0926 16:49:43.268080 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:43.268240 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:43535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260926 16:49:43.269899 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.269417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:43535 (local address 127.0.0.1:59206)
0926 16:49:43.269640 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:43.269745 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:43535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260926 16:49:43.270215 32678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:43535: Network error: Client connection negotiation failed: client connection to 127.9.238.252:43535: connect: Connection refused (error 111)
I20260926 16:49:43.273751 32752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } attempt: 3
I20260926 16:49:43.275187 32756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.274574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:43535 (local address 127.0.0.1:59210)
0926 16:49:43.274829 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:43.274963 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:43535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260926 16:49:43.276573 32756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.276052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:43535 (local address 127.0.0.1:59212)
0926 16:49:43.276327 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:43.276425 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:43535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
W20260926 16:49:43.276865 32752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:43535: Network error: Client connection negotiation failed: client connection to 127.9.238.252:43535: connect: Connection refused (error 111)
I20260926 16:49:43.376628 32678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } attempt: 3
I20260926 16:49:43.378258 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.377597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:43535 (local address 127.0.0.1:59228)
0926 16:49:43.377880 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:43.378022 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:43535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260926 16:49:43.379779 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.379269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:43535 (local address 127.0.0.1:59234)
0926 16:49:43.379481 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:43.379637 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:43535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260926 16:49:43.380102 32678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:43535: Network error: Client connection negotiation failed: client connection to 127.9.238.252:43535: connect: Connection refused (error 111)
W20260926 16:49:43.419310 32760 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 16:49:43.419822 32760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:49:43.419929 32760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:49:43.420081 32760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 16:49:43.420171 32752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } attempt: 4
I20260926 16:49:43.421651 32756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.421053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:43535 (local address 127.0.0.1:59246)
0926 16:49:43.421322 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:43.421446 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:43535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 16:49:43.422758 32756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.422418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:43535 (local address 127.0.0.1:59254)
0926 16:49:43.422566 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:43.422649 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:43535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 16:49:43.423079 32752 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:43535: Network error: Client connection negotiation failed: client connection to 127.9.238.252:43535: connect: Connection refused (error 111)
W20260926 16:49:43.428231 32760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:49:43.428342 32760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:49:43.428432 32760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:49:43.428506 32760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:49:43.428591 32760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:49:43.439363 32760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:40099,127.9.238.253:34035,127.9.238.252:43535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.252.keytab
--principal=kudu/127.9.238.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.252:43535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.238.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:49:43.440860 32760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:49:43.442199 32760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:49:43.456634 32766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:43.457209 32765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:43.457412 300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:49:43.458882 32760 server_base.cc:1034] running on GCE node
I20260926 16:49:43.459829 32760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:49:43.461417 32760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:49:43.462678 32760 hybrid_clock.cc:648] HybridClock initialized: now 1790441383462583 us; error 76 us; skew 500 ppm
I20260926 16:49:43.463065 32760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:49:43 dist-test-slave-wt2j krb5kdc[32598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441383, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:49:43.466569 32760 init.cc:377] Logged in from keytab as kudu/127.9.238.252@KRBTEST.COM (short username kudu)
I20260926 16:49:43.468284 32760 webserver.cc:460] Webserver started at http://127.9.238.252:38967/ using document root <none> and password file <none>
I20260926 16:49:43.468977 32760 fs_manager.cc:359] Metadata directory not provided
I20260926 16:49:43.469070 32760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:49:43.469357 32760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:49:43.472445 32760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-2/data/instance:
uuid: "d69adc3718694e278dfeb14e9a88de88"
format_stamp: "Formatted at 2026-09-26 16:49:43 on dist-test-slave-wt2j"
server_key: "284085e3ca56af50a470bad420c43d07"
server_key_iv: "55b9caa07c8732261903fee1a7ccf811"
server_key_version: "encryptionkey@0"
I20260926 16:49:43.473220 32760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-2/wal/instance:
uuid: "d69adc3718694e278dfeb14e9a88de88"
format_stamp: "Formatted at 2026-09-26 16:49:43 on dist-test-slave-wt2j"
server_key: "284085e3ca56af50a470bad420c43d07"
server_key_iv: "55b9caa07c8732261903fee1a7ccf811"
server_key_version: "encryptionkey@0"
I20260926 16:49:43.479218 32760 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260926 16:49:43.483678 307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:49:43.485433 32760 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:49:43.485673 32760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-2/wal
uuid: "d69adc3718694e278dfeb14e9a88de88"
format_stamp: "Formatted at 2026-09-26 16:49:43 on dist-test-slave-wt2j"
server_key: "284085e3ca56af50a470bad420c43d07"
server_key_iv: "55b9caa07c8732261903fee1a7ccf811"
server_key_version: "encryptionkey@0"
I20260926 16:49:43.485930 32760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:49:43.519470 32760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:49:43.524121 32760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:49:43.524431 32760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:49:43.546517 32760 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.252:43535
I20260926 16:49:43.546579 358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.252:43535 every 8 connection(s)
I20260926 16:49:43.548159 32760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-2/data/info.pb
I20260926 16:49:43.551182 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 32760
I20260926 16:49:43.551385 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790440988577176-10171-0/minicluster-data/master-2/wal/instance
I20260926 16:49:43.551847 10171 external_mini_cluster.cc:1428] Setting key 026aafc9e07c857a8e5a90fe0aee172d
I20260926 16:49:43.554095 359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:49:43.557412 32678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } attempt: 4
I20260926 16:49:43.565622 359 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40099 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:49:43 dist-test-slave-wt2j krb5kdc[32598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441382, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for kudu/127.9.238.252@KRBTEST.COM
Sep 26 16:49:43 dist-test-slave-wt2j krb5kdc[32598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.252@KRBTEST.COM
I20260926 16:49:43.582363 32680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.558126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:43535 (local address 127.0.0.1:59274)
0926 16:49:43.558426 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:43.558446 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 16:49:43.558563 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:49:43.559714 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:49:43.559721 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:49:43.560293 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:49:43.560516 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:43.560528 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.563408 (+ 2880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.563420 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:43.565201 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:43.565234 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.566731 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.566738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:43.566962 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:43.567920 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:49:43.567943 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:49:43.572173 (+ 4230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:49:43.578388 (+ 6215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:43.578393 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:43.578396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:43.579003 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:43.579750 (+ 747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:43.579756 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:43.579760 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:43.580004 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:43.581032 (+ 1028us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:49:43.581038 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:49:43.581268 (+ 230us) client_negotiation.cc:769] Sending connection context
0926 16:49:43.581445 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 16:49:43.581864 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"spinlock_wait_cycles":7040}
Sep 26 16:49:43 dist-test-slave-wt2j krb5kdc[32598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441383, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.252@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:49:43.584009 362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.554124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59262 (local address 127.9.238.252:43535)
0926 16:49:43.555234 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:43.555255 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:43.555326 (+ 71us) server_negotiation.cc:407] Connection header received
0926 16:49:43.558202 (+ 2876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:43.558239 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:43.559045 (+ 806us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:43.559624 (+ 579us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:43.561875 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.563491 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.565526 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.566500 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.566729 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:43.573126 (+ 6397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:43.573132 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:43.573135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:43.573178 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:43.578097 (+ 4919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:43.579132 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:43.579144 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:43.579156 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:43.579502 (+ 346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:43.580060 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:43.580066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:43.580069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:43.580845 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:43.581194 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:43.581694 (+ 500us) server_negotiation.cc:299] Negotiation successful
0926 16:49:43.582230 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":129,"server-negotiator.queue_time_us":797,"thread_start_us":462,"threads_started":1}
I20260926 16:49:43.584043 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.558276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59274 (local address 127.9.238.252:43535)
0926 16:49:43.559295 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:43.559301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:43.559327 (+ 26us) server_negotiation.cc:407] Connection header received
0926 16:49:43.559394 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:43.559399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:43.559492 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:43.560395 (+ 903us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:43.560833 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.563167 (+ 2334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.565417 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.566501 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.567352 (+ 851us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:43.572625 (+ 5273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:43.572640 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:43.572662 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:43.572735 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:43.578217 (+ 5482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:43.579151 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:43.579157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:43.579161 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:43.579503 (+ 342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:43.580136 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:43.580144 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:43.580148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:43.580839 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:43.582289 (+ 1450us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:43.582409 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 16:49:43.582634 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":843,"spinlock_wait_cycles":15232,"thread_start_us":529,"threads_started":1}
I20260926 16:49:43.593300 364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.569349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47238 (local address 127.9.238.254:40099)
0926 16:49:43.571606 (+ 2257us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:43.571613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:43.572311 (+ 698us) server_negotiation.cc:407] Connection header received
0926 16:49:43.572399 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:43.572405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:43.572490 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:43.572629 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:43.574800 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.575903 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.578695 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.579488 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.579696 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:43.584831 (+ 5135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:43.584838 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:43.584841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:43.584880 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:43.588861 (+ 3981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:43.590070 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:43.590077 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:43.590081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:43.590283 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:43.590923 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:43.590928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:43.590932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:43.591488 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:43.591725 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:43.592485 (+ 760us) server_negotiation.cc:299] Negotiation successful
0926 16:49:43.592785 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":2020,"spinlock_wait_cycles":1792,"thread_start_us":1778,"threads_started":1}
I20260926 16:49:43.593356 365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.569487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:40099 (local address 127.0.0.1:47238)
0926 16:49:43.571918 (+ 2431us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:43.571974 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 16:49:43.572165 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:49:43.572717 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:49:43.572727 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:49:43.573973 (+ 1246us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:49:43.574588 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:43.574607 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.576144 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.576153 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:43.578513 (+ 2360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:43.578531 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.579875 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.579883 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:43.580247 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:43.581122 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:49:43.581150 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:49:43.584577 (+ 3427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:49:43.589064 (+ 4487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:43.589080 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:43.589090 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:43.589911 (+ 821us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:43.590459 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:43.590465 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:43.590469 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:43.590687 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:43.591719 (+ 1032us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:49:43.591732 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:49:43.592018 (+ 286us) client_negotiation.cc:769] Sending connection context
0926 16:49:43.592354 (+ 336us) client_negotiation.cc:240] Negotiation successful
0926 16:49:43.592789 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2124,"thread_start_us":1922,"threads_started":1}
I20260926 16:49:43.593636 32678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c: Bootstrap starting.
I20260926 16:49:43.595476 359 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.253" port: 34035 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:49:43.597975 32678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c: Neither blocks nor log segments found. Creating new log.
I20260926 16:49:43.599426 32678 log.cc:824] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c: Log is configured to *not* fsync() on all Append() calls
Sep 26 16:49:43 dist-test-slave-wt2j krb5kdc[32598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
Sep 26 16:49:43 dist-test-slave-wt2j krb5kdc[32598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441383, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.252@KRBTEST.COM for kudu/127.9.238.253@KRBTEST.COM
I20260926 16:49:43.607213 32678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c: No bootstrap required, opened a new log
I20260926 16:49:43.611361 366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.588737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47244 (local address 127.9.238.254:40099)
0926 16:49:43.590302 (+ 1565us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:43.590312 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:43.590345 (+ 33us) server_negotiation.cc:407] Connection header received
0926 16:49:43.590433 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:43.590439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:43.590541 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:43.590873 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:43.592794 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.594389 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.596458 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.597193 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.597386 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:43.603469 (+ 6083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:43.603483 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:43.603487 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:43.603535 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:43.607411 (+ 3876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:43.608651 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:43.608659 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:43.608664 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:43.608843 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:43.609417 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:43.609427 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:43.609431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:43.609864 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:43.610155 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:43.610624 (+ 469us) server_negotiation.cc:299] Negotiation successful
0926 16:49:43.610893 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":1333,"thread_start_us":1045,"threads_started":1}
I20260926 16:49:43.614867 32758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.596745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41180 (local address 127.9.238.253:34035)
0926 16:49:43.596993 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:43.597003 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:43.597065 (+ 62us) server_negotiation.cc:407] Connection header received
0926 16:49:43.597301 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:43.597307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:43.597404 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:43.597511 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:43.598757 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.599886 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.602000 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.602914 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.603092 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:43.606965 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:43.606972 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:43.606976 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:43.607023 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:43.610912 (+ 3889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:43.612117 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:43.612127 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:43.612131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:43.612339 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:43.612926 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:43.612931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:43.612935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:43.613320 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:43.613493 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:43.614060 (+ 567us) server_negotiation.cc:299] Negotiation successful
0926 16:49:43.614406 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 16:49:43.615875 365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.596658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:34035 (local address 127.0.0.1:41180)
0926 16:49:43.596985 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:43.597008 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:49:43.597134 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:49:43.597703 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:49:43.597720 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:49:43.598294 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:49:43.598533 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:43.598547 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.600116 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.600124 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:43.601793 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:43.601829 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.603145 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.603154 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:43.603419 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:43.604247 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:49:43.604279 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:49:43.606678 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:49:43.611133 (+ 4455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:43.611139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:43.611143 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:43.611889 (+ 746us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:43.612498 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:43.612503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:43.612509 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:43.612721 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:43.613470 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:49:43.613476 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:49:43.613689 (+ 213us) client_negotiation.cc:769] Sending connection context
0926 16:49:43.615094 (+ 1405us) client_negotiation.cc:240] Negotiation successful
0926 16:49:43.615463 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260926 16:49:43.617199 359 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:49:43.619220 32678 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaf040a5d974d408fafd27446e03b3c" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40099 } } peers { permanent_uuid: "deb8a9fdd2e741c780e3c1d259870022" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 34035 } } peers { permanent_uuid: "d69adc3718694e278dfeb14e9a88de88" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } }
I20260926 16:49:43.619633 32678 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbaf040a5d974d408fafd27446e03b3c, State: Initialized, Role: FOLLOWER
I20260926 16:49:43.620206 32678 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaf040a5d974d408fafd27446e03b3c" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40099 } } peers { permanent_uuid: "deb8a9fdd2e741c780e3c1d259870022" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 34035 } } peers { permanent_uuid: "d69adc3718694e278dfeb14e9a88de88" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } }
Sep 26 16:49:43 dist-test-slave-wt2j krb5kdc[32598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.253@KRBTEST.COM
Sep 26 16:49:43 dist-test-slave-wt2j krb5kdc[32598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441383, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.252@KRBTEST.COM for kudu/127.9.238.252@KRBTEST.COM
I20260926 16:49:43.627427 369 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaf040a5d974d408fafd27446e03b3c" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40099 } } peers { permanent_uuid: "deb8a9fdd2e741c780e3c1d259870022" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 34035 } } peers { permanent_uuid: "d69adc3718694e278dfeb14e9a88de88" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } } }
I20260926 16:49:43.627904 369 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:49:43.629750 32678 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:49:43.637266 365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.618029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:43535 (local address 127.0.0.1:59286)
0926 16:49:43.618337 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:43.618359 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:49:43.618469 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:49:43.619912 (+ 1443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:49:43.619918 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:49:43.620466 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:49:43.620711 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:43.620724 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.622406 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.622413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:43.623497 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:43.623510 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.624358 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.624364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:43.624692 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:43.625435 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:49:43.625455 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:49:43.628726 (+ 3271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:49:43.633984 (+ 5258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:43.633988 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:43.633992 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:43.634427 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:43.634931 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:43.634936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:43.634939 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:43.635109 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:43.635822 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:49:43.635828 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:49:43.636018 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 16:49:43.636221 (+ 203us) client_negotiation.cc:240] Negotiation successful
0926 16:49:43.636674 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":24}
I20260926 16:49:43.638195 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.618202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59286 (local address 127.9.238.252:43535)
0926 16:49:43.618449 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:43.618457 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:43.618481 (+ 24us) server_negotiation.cc:407] Connection header received
0926 16:49:43.618604 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:43.618610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:43.618678 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:43.618800 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:43.620899 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.622185 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.623696 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.624190 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.624348 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:43.629031 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:43.629040 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:43.629048 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:43.629096 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:43.633726 (+ 4630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:43.634571 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:43.634579 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:43.634583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:43.634774 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:43.635220 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:43.635223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:43.635225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:43.635690 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:43.635879 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:43.637706 (+ 1827us) server_negotiation.cc:299] Negotiation successful
0926 16:49:43.637905 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":60}
I20260926 16:49:43.639786 368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.614014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41196 (local address 127.9.238.253:34035)
0926 16:49:43.614870 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:43.614876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:43.614900 (+ 24us) server_negotiation.cc:407] Connection header received
0926 16:49:43.614985 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:43.614993 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:43.615078 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:43.617066 (+ 1988us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:43.617340 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.618657 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.620422 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.621070 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.621226 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:43.629188 (+ 7962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:43.629199 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:43.629204 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:43.629275 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:43.633767 (+ 4492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:43.635214 (+ 1447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:43.635225 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:43.635232 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:43.635495 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:43.636126 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:43.636134 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:43.636137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:43.636572 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:43.636921 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:43.638759 (+ 1838us) server_negotiation.cc:299] Negotiation successful
0926 16:49:43.639140 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":381,"threads_started":1}
I20260926 16:49:43.646888 359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d69adc3718694e278dfeb14e9a88de88: Bootstrap starting.
I20260926 16:49:43.648125 10171 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 16:49:43.650995 380 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:49:43.651185 380 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:49:43.652261 32653 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "cbaf040a5d974d408fafd27446e03b3c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47244
I20260926 16:49:43.653100 32653 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 16:49:43.653479 32653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:49:43.653892 359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d69adc3718694e278dfeb14e9a88de88: Neither blocks nor log segments found. Creating new log.
I20260926 16:49:43.655247 359 log.cc:824] T 00000000000000000000000000000000 P d69adc3718694e278dfeb14e9a88de88: Log is configured to *not* fsync() on all Append() calls
I20260926 16:49:43.657486 32653 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaf040a5d974d408fafd27446e03b3c" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40099 } } peers { permanent_uuid: "deb8a9fdd2e741c780e3c1d259870022" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 34035 } } peers { permanent_uuid: "d69adc3718694e278dfeb14e9a88de88" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } }
I20260926 16:49:43.661674 32653 leader_election.cc:288] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c [CANDIDATE]: Term 1 election: Requested vote from peers deb8a9fdd2e741c780e3c1d259870022 (127.9.238.253:34035), d69adc3718694e278dfeb14e9a88de88 (127.9.238.252:43535)
I20260926 16:49:43.652966 32611 master_runner.cc:186] Error getting master registration for 127.9.238.253:34035: OK, instance_id { permanent_uuid: "deb8a9fdd2e741c780e3c1d259870022" instance_seqno: 1790441382967621 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 16:49:43.667289 32611 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 16:49:43.670261 359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d69adc3718694e278dfeb14e9a88de88: No bootstrap required, opened a new log
I20260926 16:49:43.694765 359 raft_consensus.cc:348] T 00000000000000000000000000000000 P d69adc3718694e278dfeb14e9a88de88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaf040a5d974d408fafd27446e03b3c" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40099 } } peers { permanent_uuid: "deb8a9fdd2e741c780e3c1d259870022" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 34035 } } peers { permanent_uuid: "d69adc3718694e278dfeb14e9a88de88" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } }
I20260926 16:49:43.695431 359 raft_consensus.cc:729] T 00000000000000000000000000000000 P d69adc3718694e278dfeb14e9a88de88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d69adc3718694e278dfeb14e9a88de88, State: Initialized, Role: FOLLOWER
I20260926 16:49:43.696197 359 consensus_queue.cc:260] T 00000000000000000000000000000000 P d69adc3718694e278dfeb14e9a88de88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaf040a5d974d408fafd27446e03b3c" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40099 } } peers { permanent_uuid: "deb8a9fdd2e741c780e3c1d259870022" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 34035 } } peers { permanent_uuid: "d69adc3718694e278dfeb14e9a88de88" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } }
I20260926 16:49:43.703598 390 sys_catalog.cc:455] T 00000000000000000000000000000000 P d69adc3718694e278dfeb14e9a88de88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaf040a5d974d408fafd27446e03b3c" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40099 } } peers { permanent_uuid: "deb8a9fdd2e741c780e3c1d259870022" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 34035 } } peers { permanent_uuid: "d69adc3718694e278dfeb14e9a88de88" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } } }
I20260926 16:49:43.704211 390 sys_catalog.cc:458] T 00000000000000000000000000000000 P d69adc3718694e278dfeb14e9a88de88 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:49:43.706949 359 sys_catalog.cc:564] T 00000000000000000000000000000000 P d69adc3718694e278dfeb14e9a88de88 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:49:43.706982 334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbaf040a5d974d408fafd27446e03b3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d69adc3718694e278dfeb14e9a88de88"
I20260926 16:49:43.708025 334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d69adc3718694e278dfeb14e9a88de88 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:49:43.712677 368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.671024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41204 (local address 127.9.238.253:34035)
0926 16:49:43.671236 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:43.671242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:43.688110 (+ 16868us) server_negotiation.cc:407] Connection header received
0926 16:49:43.688470 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:43.688478 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:43.688584 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:43.688666 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:43.696353 (+ 7687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.697445 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.700157 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.700829 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.700937 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:43.705069 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:43.705077 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:43.705082 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:43.705137 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:43.709085 (+ 3948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:43.710167 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:43.710172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:43.710176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:43.710325 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:43.710813 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:43.710818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:43.710821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:43.711200 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:43.711439 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:43.712060 (+ 621us) server_negotiation.cc:299] Negotiation successful
0926 16:49:43.712305 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 16:49:43.714699 334 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d69adc3718694e278dfeb14e9a88de88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbaf040a5d974d408fafd27446e03b3c in term 1.
I20260926 16:49:43.716307 32630 leader_election.cc:302] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbaf040a5d974d408fafd27446e03b3c, d69adc3718694e278dfeb14e9a88de88; no voters:
I20260926 16:49:43.717130 369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:49:43.722291 32752 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } attempt: 5
I20260926 16:49:43.722962 369 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c [term 1 LEADER]: Becoming Leader. State: Replica: cbaf040a5d974d408fafd27446e03b3c, State: Running, Role: LEADER
I20260926 16:49:43.723769 369 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaf040a5d974d408fafd27446e03b3c" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40099 } } peers { permanent_uuid: "deb8a9fdd2e741c780e3c1d259870022" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 34035 } } peers { permanent_uuid: "d69adc3718694e278dfeb14e9a88de88" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } }
I20260926 16:49:43.727303 366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.671461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47248 (local address 127.9.238.254:40099)
0926 16:49:43.671811 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:43.671819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:43.676427 (+ 4608us) server_negotiation.cc:407] Connection header received
0926 16:49:43.683860 (+ 7433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:43.683870 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:43.684037 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:43.684148 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:43.687349 (+ 3201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.688720 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.694059 (+ 5339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.694932 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.695127 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:43.717122 (+ 21995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:43.717129 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:43.717134 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:43.717181 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:43.722430 (+ 5249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:43.723713 (+ 1283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:43.723721 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:43.723724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:43.723934 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:43.724514 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:43.724520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:43.724523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:43.724965 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:43.725272 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:43.726541 (+ 1269us) server_negotiation.cc:299] Negotiation successful
0926 16:49:43.726825 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
Sep 26 16:49:43 dist-test-slave-wt2j krb5kdc[32598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441382, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.253@KRBTEST.COM for kudu/127.9.238.252@KRBTEST.COM
I20260926 16:49:43.770105 362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.724018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59304 (local address 127.9.238.252:43535)
0926 16:49:43.724236 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:43.724243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:43.724262 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:49:43.724319 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:43.724325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:43.724415 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:43.724510 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:43.725802 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.726848 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.730582 (+ 3734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.731524 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.731705 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:43.742716 (+ 11011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:43.742730 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:43.742738 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:43.742826 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:43.751386 (+ 8560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:43.758910 (+ 7524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:43.758924 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:43.758931 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:43.759460 (+ 529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:43.761884 (+ 2424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:43.761894 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:43.761900 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:43.763052 (+ 1152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:43.763456 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:43.766608 (+ 3152us) server_negotiation.cc:299] Negotiation successful
0926 16:49:43.767391 (+ 783us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 16:49:43.774282 32756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.723133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:43535 (local address 127.0.0.1:59304)
0926 16:49:43.723638 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:43.723661 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:49:43.723777 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:49:43.724756 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:49:43.724760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:49:43.725356 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:49:43.725584 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:43.725596 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.727213 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.727220 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:43.729201 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:43.729226 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.732160 (+ 2934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.732171 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:43.732768 (+ 597us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:43.734209 (+ 1441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:49:43.734270 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:49:43.741953 (+ 7683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:49:43.752710 (+ 10757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:43.752724 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:43.752732 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:43.757979 (+ 5247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:43.760336 (+ 2357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:43.760437 (+ 101us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:43.760447 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:43.761392 (+ 945us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:43.763811 (+ 2419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:49:43.763921 (+ 110us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:49:43.765263 (+ 1342us) client_negotiation.cc:769] Sending connection context
0926 16:49:43.768157 (+ 2894us) client_negotiation.cc:240] Negotiation successful
0926 16:49:43.771636 (+ 3479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306}
W20260926 16:49:43.779968 313 tablet.cc:2369] T 00000000000000000000000000000000 P d69adc3718694e278dfeb14e9a88de88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:49:43.786499 32752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deb8a9fdd2e741c780e3c1d259870022: Bootstrap starting.
I20260926 16:49:43.794891 32752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P deb8a9fdd2e741c780e3c1d259870022: Neither blocks nor log segments found. Creating new log.
I20260926 16:49:43.796842 32752 log.cc:824] T 00000000000000000000000000000000 P deb8a9fdd2e741c780e3c1d259870022: Log is configured to *not* fsync() on all Append() calls
I20260926 16:49:43.801448 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:43.689110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59294 (local address 127.9.238.252:43535)
0926 16:49:43.689393 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:43.689400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:43.695399 (+ 5999us) server_negotiation.cc:407] Connection header received
0926 16:49:43.696014 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:43.696021 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:43.696123 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:43.696267 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:43.717151 (+ 20884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.718674 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.752887 (+ 34213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:43.754010 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:43.754156 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:43.791343 (+ 37187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:43.791354 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:43.791363 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:43.791429 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:43.795694 (+ 4265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:43.796915 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:43.796921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:43.796925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:43.797115 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:43.798074 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:43.798084 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:43.798090 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:43.798645 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:43.799059 (+ 414us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:43.800430 (+ 1371us) server_negotiation.cc:299] Negotiation successful
0926 16:49:43.800826 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260926 16:49:43.807200 393 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbaf040a5d974d408fafd27446e03b3c. Latest consensus state: current_term: 1 leader_uuid: "cbaf040a5d974d408fafd27446e03b3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaf040a5d974d408fafd27446e03b3c" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40099 } } peers { permanent_uuid: "deb8a9fdd2e741c780e3c1d259870022" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 34035 } } peers { permanent_uuid: "d69adc3718694e278dfeb14e9a88de88" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } } }
I20260926 16:49:43.809090 393 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:43.813879 32752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deb8a9fdd2e741c780e3c1d259870022: No bootstrap required, opened a new log
I20260926 16:49:43.810549 399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:49:43.819056 399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:49:43.831985 32752 raft_consensus.cc:348] T 00000000000000000000000000000000 P deb8a9fdd2e741c780e3c1d259870022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaf040a5d974d408fafd27446e03b3c" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40099 } } peers { permanent_uuid: "deb8a9fdd2e741c780e3c1d259870022" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 34035 } } peers { permanent_uuid: "d69adc3718694e278dfeb14e9a88de88" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } }
I20260926 16:49:43.832706 32752 raft_consensus.cc:729] T 00000000000000000000000000000000 P deb8a9fdd2e741c780e3c1d259870022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deb8a9fdd2e741c780e3c1d259870022, State: Initialized, Role: FOLLOWER
I20260926 16:49:43.833626 32752 consensus_queue.cc:260] T 00000000000000000000000000000000 P deb8a9fdd2e741c780e3c1d259870022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaf040a5d974d408fafd27446e03b3c" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40099 } } peers { permanent_uuid: "deb8a9fdd2e741c780e3c1d259870022" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 34035 } } peers { permanent_uuid: "d69adc3718694e278dfeb14e9a88de88" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } }
I20260926 16:49:43.835440 403 sys_catalog.cc:455] T 00000000000000000000000000000000 P deb8a9fdd2e741c780e3c1d259870022 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaf040a5d974d408fafd27446e03b3c" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40099 } } peers { permanent_uuid: "deb8a9fdd2e741c780e3c1d259870022" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 34035 } } peers { permanent_uuid: "d69adc3718694e278dfeb14e9a88de88" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } } }
I20260926 16:49:43.836349 403 sys_catalog.cc:458] T 00000000000000000000000000000000 P deb8a9fdd2e741c780e3c1d259870022 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:49:43.838430 32752 sys_catalog.cc:564] T 00000000000000000000000000000000 P deb8a9fdd2e741c780e3c1d259870022 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:49:43.838544 32727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbaf040a5d974d408fafd27446e03b3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "deb8a9fdd2e741c780e3c1d259870022"
I20260926 16:49:43.839246 32727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P deb8a9fdd2e741c780e3c1d259870022 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:49:43.843160 334 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d69adc3718694e278dfeb14e9a88de88 [term 1 FOLLOWER]: Refusing update from remote peer cbaf040a5d974d408fafd27446e03b3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:49:43.843664 32727 raft_consensus.cc:2454] T 00000000000000000000000000000000 P deb8a9fdd2e741c780e3c1d259870022 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbaf040a5d974d408fafd27446e03b3c in term 1.
I20260926 16:49:43.845983 393 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d69adc3718694e278dfeb14e9a88de88" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:49:43.857219 32727 raft_consensus.cc:1261] T 00000000000000000000000000000000 P deb8a9fdd2e741c780e3c1d259870022 [term 1 FOLLOWER]: Refusing update from remote peer cbaf040a5d974d408fafd27446e03b3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:49:43.858595 393 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "deb8a9fdd2e741c780e3c1d259870022" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 34035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:49:43.869246 403 sys_catalog.cc:455] T 00000000000000000000000000000000 P deb8a9fdd2e741c780e3c1d259870022 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbaf040a5d974d408fafd27446e03b3c. Latest consensus state: current_term: 1 leader_uuid: "cbaf040a5d974d408fafd27446e03b3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaf040a5d974d408fafd27446e03b3c" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40099 } } peers { permanent_uuid: "deb8a9fdd2e741c780e3c1d259870022" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 34035 } } peers { permanent_uuid: "d69adc3718694e278dfeb14e9a88de88" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } } }
I20260926 16:49:43.869694 403 sys_catalog.cc:458] T 00000000000000000000000000000000 P deb8a9fdd2e741c780e3c1d259870022 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 16:49:43.876302 32632 tablet.cc:2369] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:49:43.878988 401 mvcc.cc:205] Tried to move back new op lower bound from 7333647908203646976 to 7333647907738951680. Current Snapshot: MvccSnapshot[applied={T|T < 7333647908203646976}]
I20260926 16:49:43.890125 399 catalog_manager.cc:1347] Generated new cluster ID: e0cf31bcc6e9417088cc319ee4ff0883
I20260926 16:49:43.890265 399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:49:43.891888 369 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cbaf040a5d974d408fafd27446e03b3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaf040a5d974d408fafd27446e03b3c" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40099 } } peers { permanent_uuid: "deb8a9fdd2e741c780e3c1d259870022" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 34035 } } peers { permanent_uuid: "d69adc3718694e278dfeb14e9a88de88" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } } }
I20260926 16:49:43.892246 369 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:43.891953 390 sys_catalog.cc:455] T 00000000000000000000000000000000 P d69adc3718694e278dfeb14e9a88de88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbaf040a5d974d408fafd27446e03b3c. Latest consensus state: current_term: 1 leader_uuid: "cbaf040a5d974d408fafd27446e03b3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaf040a5d974d408fafd27446e03b3c" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40099 } } peers { permanent_uuid: "deb8a9fdd2e741c780e3c1d259870022" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 34035 } } peers { permanent_uuid: "d69adc3718694e278dfeb14e9a88de88" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } } }
I20260926 16:49:43.892513 390 sys_catalog.cc:458] T 00000000000000000000000000000000 P d69adc3718694e278dfeb14e9a88de88 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:49:43.893503 403 sys_catalog.cc:455] T 00000000000000000000000000000000 P deb8a9fdd2e741c780e3c1d259870022 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cbaf040a5d974d408fafd27446e03b3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaf040a5d974d408fafd27446e03b3c" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40099 } } peers { permanent_uuid: "deb8a9fdd2e741c780e3c1d259870022" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 34035 } } peers { permanent_uuid: "d69adc3718694e278dfeb14e9a88de88" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } } }
I20260926 16:49:43.893879 403 sys_catalog.cc:458] T 00000000000000000000000000000000 P deb8a9fdd2e741c780e3c1d259870022 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:49:43.902144 393 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cbaf040a5d974d408fafd27446e03b3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaf040a5d974d408fafd27446e03b3c" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40099 } } peers { permanent_uuid: "deb8a9fdd2e741c780e3c1d259870022" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 34035 } } peers { permanent_uuid: "d69adc3718694e278dfeb14e9a88de88" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } } }
I20260926 16:49:43.902688 393 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:43.905267 406 mvcc.cc:205] Tried to move back new op lower bound from 7333647908203646976 to 7333647907738951680. Current Snapshot: MvccSnapshot[applied={T|T < 7333647908203646976}]
I20260926 16:49:43.906352 390 sys_catalog.cc:455] T 00000000000000000000000000000000 P d69adc3718694e278dfeb14e9a88de88 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cbaf040a5d974d408fafd27446e03b3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbaf040a5d974d408fafd27446e03b3c" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 40099 } } peers { permanent_uuid: "deb8a9fdd2e741c780e3c1d259870022" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 34035 } } peers { permanent_uuid: "d69adc3718694e278dfeb14e9a88de88" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 43535 } } }
I20260926 16:49:43.906687 390 sys_catalog.cc:458] T 00000000000000000000000000000000 P d69adc3718694e278dfeb14e9a88de88 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:49:43.924073 416 catalog_manager.cc:1259] Loaded cluster ID: e0cf31bcc6e9417088cc319ee4ff0883
I20260926 16:49:43.924229 416 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d69adc3718694e278dfeb14e9a88de88: loading cluster ID for follower catalog manager: success
W20260926 16:49:43.929718 416 catalog_manager.cc:1576] T 00000000000000000000000000000000 P d69adc3718694e278dfeb14e9a88de88: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260926 16:49:43.929903 416 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:49:43.923650 32760 master_runner.cc:186] Error getting master registration for 127.9.238.253:34035: OK, instance_id { permanent_uuid: "deb8a9fdd2e741c780e3c1d259870022" instance_seqno: 1790441382967621 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 16:49:43.940840 32760 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 16:49:43.946668 399 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:49:43.948935 399 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:49:43.986198 399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbaf040a5d974d408fafd27446e03b3c: Generated new TSK 0
I20260926 16:49:43.987861 399 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 16:49:44.070864 428 catalog_manager.cc:1259] Loaded cluster ID: e0cf31bcc6e9417088cc319ee4ff0883
I20260926 16:49:44.071063 428 catalog_manager.cc:1552] T 00000000000000000000000000000000 P deb8a9fdd2e741c780e3c1d259870022: loading cluster ID for follower catalog manager: success
I20260926 16:49:44.079455 428 catalog_manager.cc:1574] T 00000000000000000000000000000000 P deb8a9fdd2e741c780e3c1d259870022: acquiring CA information for follower catalog manager: success
I20260926 16:49:44.082194 428 catalog_manager.cc:1602] T 00000000000000000000000000000000 P deb8a9fdd2e741c780e3c1d259870022: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 16:49:44.148502 366 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:49:44.130402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47254 (local address 127.9.238.254:40099)
0926 16:49:44.130639 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:44.130646 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:44.145364 (+ 14718us) server_negotiation.cc:407] Connection header received
0926 16:49:44.146324 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:44.146333 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:44.146500 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:44.146613 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:44.148140 (+ 1527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47254: BlockingRecv error: recv got EOF from 127.0.0.1:47254 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260926 16:49:44.152168 368 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:49:44.139451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41208 (local address 127.9.238.253:34035)
0926 16:49:44.139766 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:44.139774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:44.149509 (+ 9735us) server_negotiation.cc:407] Connection header received
0926 16:49:44.149858 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:44.149864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:44.150039 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:44.150169 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:44.151859 (+ 1690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41208: BlockingRecv error: recv got EOF from 127.0.0.1:41208 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 16:49:44.153446 363 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:49:44.150422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59310 (local address 127.9.238.252:43535)
0926 16:49:44.150649 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:44.150656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:44.151199 (+ 543us) server_negotiation.cc:407] Connection header received
0926 16:49:44.151496 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:44.151503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:44.151631 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:44.151722 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:44.153078 (+ 1356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59310: BlockingRecv error: recv got EOF from 127.0.0.1:59310 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 16:49:44.214715 368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:44.197895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41216 (local address 127.9.238.253:34035)
0926 16:49:44.198143 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:44.198154 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:44.205278 (+ 7124us) server_negotiation.cc:407] Connection header received
0926 16:49:44.205972 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:44.205979 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:44.206357 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:44.206544 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:49:44.207419 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:44.208731 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:44.210944 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:44.211697 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:44.211811 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:44.212664 (+ 853us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:49:44.213628 (+ 964us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:49:44.213770 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:44.214019 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 16:49:44.214134 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260926 16:49:44.271800 363 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:49:44.261033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59316 (local address 127.9.238.252:43535)
0926 16:49:44.261293 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:44.261301 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:44.268912 (+ 7611us) server_negotiation.cc:407] Connection header received
0926 16:49:44.269673 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:44.269682 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:44.269881 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:44.269983 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:44.271400 (+ 1417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59316: BlockingRecv error: recv got EOF from 127.0.0.1:59316 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 16:49:44.323350 368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:44.306904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41228 (local address 127.9.238.253:34035)
0926 16:49:44.307187 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:44.307196 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:44.314565 (+ 7369us) server_negotiation.cc:407] Connection header received
0926 16:49:44.315237 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:44.315245 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:44.315365 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:44.315448 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:49:44.316297 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:44.317715 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:44.320385 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:44.321296 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:44.321462 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:44.322186 (+ 724us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:49:44.322410 (+ 224us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:49:44.322519 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:44.322769 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 16:49:44.322914 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
W20260926 16:49:44.338872 363 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:49:44.333177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59324 (local address 127.9.238.252:43535)
0926 16:49:44.333441 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:44.333449 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:44.336383 (+ 2934us) server_negotiation.cc:407] Connection header received
0926 16:49:44.336781 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:44.336789 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:44.337086 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:44.337204 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:44.338538 (+ 1334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59324: BlockingRecv error: recv got EOF from 127.0.0.1:59324 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 16:49:44.359948 368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:44.345718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41230 (local address 127.9.238.253:34035)
0926 16:49:44.345976 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:44.345984 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:44.352441 (+ 6457us) server_negotiation.cc:407] Connection header received
0926 16:49:44.352847 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:44.352856 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:44.352996 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:44.353071 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:49:44.353826 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:44.355121 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:44.357262 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:44.358010 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:44.358139 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:44.358845 (+ 706us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:49:44.359067 (+ 222us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:49:44.359139 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:44.359388 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 16:49:44.359504 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260926 16:49:44.372354 363 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:49:44.369381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59326 (local address 127.9.238.252:43535)
0926 16:49:44.369628 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:44.369635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:44.370280 (+ 645us) server_negotiation.cc:407] Connection header received
0926 16:49:44.370629 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:44.370636 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:44.370744 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:44.370841 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:44.372063 (+ 1222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59326: BlockingRecv error: recv got EOF from 127.0.0.1:59326 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 16:49:44.390937 368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:44.381637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41238 (local address 127.9.238.253:34035)
0926 16:49:44.381982 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:44.381990 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:44.382666 (+ 676us) server_negotiation.cc:407] Connection header received
0926 16:49:44.383064 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:44.383072 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:44.383189 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:44.383270 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:49:44.384106 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:44.385496 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:44.387852 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:44.388692 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:44.388834 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:44.389747 (+ 913us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:49:44.389974 (+ 227us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:49:44.390057 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:44.390349 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 16:49:44.390492 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
W20260926 16:49:44.428743 363 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:49:44.425567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59332 (local address 127.9.238.252:43535)
0926 16:49:44.425897 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:44.425904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:44.426643 (+ 739us) server_negotiation.cc:407] Connection header received
0926 16:49:44.426969 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:44.426977 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:44.427115 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:44.427268 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:44.428443 (+ 1175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59332: BlockingRecv error: recv got EOF from 127.0.0.1:59332 (error 108)
Metrics: {"server-negotiator.queue_time_us":139}
I20260926 16:49:44.449586 368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:44.441133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41242 (local address 127.9.238.253:34035)
0926 16:49:44.441367 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:44.441376 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:44.442212 (+ 836us) server_negotiation.cc:407] Connection header received
0926 16:49:44.442624 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:44.442630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:44.442789 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:44.442929 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:49:44.443607 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:44.444794 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:44.447018 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:44.447781 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:44.447928 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:44.448628 (+ 700us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:49:44.448794 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:49:44.448904 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:44.449103 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 16:49:44.449214 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260926 16:49:44.468442 363 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:49:44.465574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59340 (local address 127.9.238.252:43535)
0926 16:49:44.465833 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:44.465840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:44.466527 (+ 687us) server_negotiation.cc:407] Connection header received
0926 16:49:44.466767 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:44.466774 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:44.466900 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:44.467035 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:44.468128 (+ 1093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59340: BlockingRecv error: recv got EOF from 127.0.0.1:59340 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 16:49:44.498775 368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:44.489812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41258 (local address 127.9.238.253:34035)
0926 16:49:44.490028 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:44.490034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:44.491184 (+ 1150us) server_negotiation.cc:407] Connection header received
0926 16:49:44.491539 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:44.491587 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:44.491708 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:44.491857 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:49:44.492527 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:44.493947 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:44.496105 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:44.496801 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:44.496932 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:44.497654 (+ 722us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:49:44.497851 (+ 197us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:49:44.497990 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:44.498195 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 16:49:44.498342 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260926 16:49:44.510603 363 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:49:44.507596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59350 (local address 127.9.238.252:43535)
0926 16:49:44.507873 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:44.507881 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:44.508578 (+ 697us) server_negotiation.cc:407] Connection header received
0926 16:49:44.508920 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:44.508926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:44.509045 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:44.509194 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:44.510318 (+ 1124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59350: BlockingRecv error: recv got EOF from 127.0.0.1:59350 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
I20260926 16:49:44.589628 368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:44.581398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41270 (local address 127.9.238.253:34035)
0926 16:49:44.581746 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:44.581754 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:44.582549 (+ 795us) server_negotiation.cc:407] Connection header received
0926 16:49:44.582833 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:44.582839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:44.582955 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:44.583080 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:49:44.583697 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:44.584918 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:44.587289 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:44.587933 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:44.588074 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:44.588664 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:49:44.588814 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:49:44.588913 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:44.589133 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 16:49:44.589223 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
W20260926 16:49:44.602581 363 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:49:44.598197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59364 (local address 127.9.238.252:43535)
0926 16:49:44.598443 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:44.598452 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:44.600948 (+ 2496us) server_negotiation.cc:407] Connection header received
0926 16:49:44.601246 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:44.601252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:44.601332 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:44.601448 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:44.602314 (+ 866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59364: BlockingRecv error: recv got EOF from 127.0.0.1:59364 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 16:49:44.677731 368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:44.671384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41286 (local address 127.9.238.253:34035)
0926 16:49:44.671818 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:44.671828 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:44.672085 (+ 257us) server_negotiation.cc:407] Connection header received
0926 16:49:44.672468 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:44.672473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:44.672575 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:44.672693 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:49:44.673194 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:44.674082 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:44.675635 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:44.676163 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:44.676331 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:44.676808 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:49:44.676954 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:49:44.677065 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:44.677236 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 16:49:44.677337 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
W20260926 16:49:44.687852 363 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:49:44.685538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59378 (local address 127.9.238.252:43535)
0926 16:49:44.685799 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:44.685807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:44.686041 (+ 234us) server_negotiation.cc:407] Connection header received
0926 16:49:44.686328 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:44.686341 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:44.686428 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:44.686552 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:44.687469 (+ 917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59378: BlockingRecv error: recv got EOF from 127.0.0.1:59378 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
I20260926 16:49:44.826735 368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:44.820898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41290 (local address 127.9.238.253:34035)
0926 16:49:44.821213 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:44.821221 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:44.821613 (+ 392us) server_negotiation.cc:407] Connection header received
0926 16:49:44.821876 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:44.821881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:44.821958 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:44.822064 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:49:44.822582 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:44.823366 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:44.824892 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:44.825340 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:44.825487 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:44.825929 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:49:44.826077 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:49:44.826191 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:44.826335 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 16:49:44.826422 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
W20260926 16:49:44.836761 363 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:49:44.834487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59384 (local address 127.9.238.252:43535)
0926 16:49:44.834796 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:44.834804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:44.835054 (+ 250us) server_negotiation.cc:407] Connection header received
0926 16:49:44.835349 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:44.835354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:44.835450 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:44.835601 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:44.836497 (+ 896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59384: BlockingRecv error: recv got EOF from 127.0.0.1:59384 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
I20260926 16:49:44.933413 416 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d69adc3718694e278dfeb14e9a88de88: acquiring CA information for follower catalog manager: success
I20260926 16:49:44.935365 416 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d69adc3718694e278dfeb14e9a88de88: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 16:49:45.104089 368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:45.098131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41300 (local address 127.9.238.253:34035)
0926 16:49:45.098547 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:45.098553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:45.098624 (+ 71us) server_negotiation.cc:407] Connection header received
0926 16:49:45.098734 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:45.098739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:45.098859 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:45.098994 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:49:45.099631 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:45.100495 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:45.102135 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:45.102569 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:45.102723 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:45.103170 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:49:45.103302 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:49:45.103410 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:45.103611 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 16:49:45.103708 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227}
I20260926 16:49:45.118139 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:45.111271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59390 (local address 127.9.238.252:43535)
0926 16:49:45.111523 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:45.111529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:45.111822 (+ 293us) server_negotiation.cc:407] Connection header received
0926 16:49:45.112056 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:45.112062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:45.112342 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:45.112551 (+ 209us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:49:45.113018 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:45.113923 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:45.115512 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:45.116029 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:45.116153 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:45.116742 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:49:45.117491 (+ 749us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:49:45.117616 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:45.117745 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 16:49:45.117857 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260926 16:49:45.135485 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:45.127125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59392 (local address 127.9.238.252:43535)
0926 16:49:45.127450 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:45.127460 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:45.128487 (+ 1027us) server_negotiation.cc:407] Connection header received
0926 16:49:45.128748 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:45.128755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:45.128873 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:45.128956 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:49:45.129941 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:45.131032 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:45.133132 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:45.133810 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:45.133953 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:45.134536 (+ 583us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:49:45.134716 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:49:45.134824 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:45.135007 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 16:49:45.135116 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 16:49:45.135767 368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:45.125910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41314 (local address 127.9.238.253:34035)
0926 16:49:45.126125 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:45.126132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:45.127993 (+ 1861us) server_negotiation.cc:407] Connection header received
0926 16:49:45.128357 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:45.128363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:45.128483 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:45.128604 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:49:45.129326 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:45.130386 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:45.133316 (+ 2930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:45.134018 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:45.134171 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:45.134762 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:49:45.134975 (+ 213us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:49:45.135077 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:45.135261 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 16:49:45.135366 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 16:49:45.136921 543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:45.126158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47270 (local address 127.9.238.254:40099)
0926 16:49:45.128155 (+ 1997us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:45.128162 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:45.128187 (+ 25us) server_negotiation.cc:407] Connection header received
0926 16:49:45.128352 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:45.128358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:45.128640 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:45.128848 (+ 208us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:49:45.129620 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:45.130802 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:45.133530 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:45.134136 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:45.134290 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:45.135145 (+ 855us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:49:45.135810 (+ 665us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:49:45.135934 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:45.136232 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 16:49:45.136343 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1819,"thread_start_us":1615,"threads_started":1}
I20260926 16:49:45.142701 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 32611
I20260926 16:49:45.144042 32673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:49:45.278602 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 32611
I20260926 16:49:45.304446 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 32685
I20260926 16:49:45.305977 32747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:49:45.434425 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 32685
I20260926 16:49:45.459780 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 32760
I20260926 16:49:45.461261 354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:49:45.574195 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 32760
2026-09-26T16:49:45Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5194 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 16:49:45 dist-test-slave-wt2j krb5kdc[550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 16:49:45 dist-test-slave-wt2j krb5kdc[550](info): set up 2 sockets
Sep 26 16:49:45 dist-test-slave-wt2j krb5kdc[550](info): commencing operation
krb5kdc: starting...
W20260926 16:49:46.745549 10171 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/krb5kdc/test-user.keytab.
Sep 26 16:49:46 dist-test-slave-wt2j krb5kdc[550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T16:49:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T16:49:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.9.238.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
Entry for principal kudu/127.9.238.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab.
I20260926 16:49:47.076683 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:32879
--webserver_interface=127.9.238.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:43901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:32879,127.9.238.253:44489,127.9.238.252:33901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 16:49:47.431095 563 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 16:49:47.431468 563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:49:47.431636 563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:49:47.431782 563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:49:47.440027 563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:49:47.440145 563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:49:47.440202 563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:49:47.440291 563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:49:47.440407 563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:49:47.450883 563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:43901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:32879,127.9.238.253:44489,127.9.238.252:33901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.254.keytab
--principal=kudu/127.9.238.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.254:32879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.9.238.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:49:47.452374 563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:49:47.453686 563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:49:47.467526 568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:47.467502 569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:47.468321 571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:49:47.468945 563 server_base.cc:1034] running on GCE node
I20260926 16:49:47.469853 563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:49:47.471470 563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:49:47.472719 563 hybrid_clock.cc:648] HybridClock initialized: now 1790441387472631 us; error 60 us; skew 500 ppm
I20260926 16:49:47.473104 563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:49:47 dist-test-slave-wt2j krb5kdc[550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441387, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:49:47.484091 563 init.cc:377] Logged in from keytab as kudu/127.9.238.254@KRBTEST.COM (short username kudu)
I20260926 16:49:47.486212 563 webserver.cc:460] Webserver started at http://127.9.238.254:36741/ using document root <none> and password file <none>
I20260926 16:49:47.487046 563 fs_manager.cc:359] Metadata directory not provided
I20260926 16:49:47.487157 563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:49:47.487622 563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:49:47.491106 563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-0/data/instance:
uuid: "b7d51ceaa460445cb2a43d70b5d415be"
format_stamp: "Formatted at 2026-09-26 16:49:47 on dist-test-slave-wt2j"
server_key: "ff735a8198b1439f77add8f6b13f8331"
server_key_iv: "5ae18e3b7a260098b26d0ce49f80a96f"
server_key_version: "encryptionkey@0"
I20260926 16:49:47.492082 563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-0/wal/instance:
uuid: "b7d51ceaa460445cb2a43d70b5d415be"
format_stamp: "Formatted at 2026-09-26 16:49:47 on dist-test-slave-wt2j"
server_key: "ff735a8198b1439f77add8f6b13f8331"
server_key_iv: "5ae18e3b7a260098b26d0ce49f80a96f"
server_key_version: "encryptionkey@0"
I20260926 16:49:47.498358 563 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260926 16:49:47.503072 578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:49:47.505026 563 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 16:49:47.505276 563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-0/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-0/wal
uuid: "b7d51ceaa460445cb2a43d70b5d415be"
format_stamp: "Formatted at 2026-09-26 16:49:47 on dist-test-slave-wt2j"
server_key: "ff735a8198b1439f77add8f6b13f8331"
server_key_iv: "5ae18e3b7a260098b26d0ce49f80a96f"
server_key_version: "encryptionkey@0"
I20260926 16:49:47.505551 563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:49:47.534704 563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:49:47.539472 563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:49:47.539831 563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:49:47.563334 563 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.254:32879
I20260926 16:49:47.563386 629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.254:32879 every 8 connection(s)
I20260926 16:49:47.564839 563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-0/data/info.pb
I20260926 16:49:47.569927 630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:49:47.571604 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 563
I20260926 16:49:47.571841 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-0/wal/instance
I20260926 16:49:47.572671 10171 external_mini_cluster.cc:1428] Setting key d55970abb29b69b55d87f2dc9b15a91b
I20260926 16:49:47.583971 630 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.254" port: 32879 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:49:47 dist-test-slave-wt2j krb5kdc[550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441387, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.238.253@KRBTEST.COM; defaulting to no policy
I20260926 16:49:47.614248 634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:47.589315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43160 (local address 127.9.238.254:32879)
0926 16:49:47.590591 (+ 1276us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:47.590603 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:47.590722 (+ 119us) server_negotiation.cc:407] Connection header received
0926 16:49:47.593166 (+ 2444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:47.593179 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:47.593623 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:47.593907 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:47.595264 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:47.597366 (+ 2102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:47.599098 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:47.600279 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:47.600547 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:47.605034 (+ 4487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:47.605044 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:47.605060 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:47.605097 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:47.608950 (+ 3853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:47.610092 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:47.610103 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:47.610110 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:47.610346 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:47.610903 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:47.610911 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:47.610916 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:47.611766 (+ 850us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:47.611996 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:47.612636 (+ 640us) server_negotiation.cc:299] Negotiation successful
0926 16:49:47.612968 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":873,"thread_start_us":597,"threads_started":1}
I20260926 16:49:47.614277 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:47.589315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:32879 (local address 127.0.0.1:43160)
0926 16:49:47.590511 (+ 1196us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:47.590572 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 16:49:47.592695 (+ 2123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:49:47.593958 (+ 1263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:49:47.593963 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:49:47.594597 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:49:47.594993 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:47.595016 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:47.597609 (+ 2593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:47.597614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:47.598899 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:47.598914 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:47.600574 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:47.600580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:47.600739 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:47.601572 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:49:47.601597 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:49:47.604704 (+ 3107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:49:47.609166 (+ 4462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:47.609181 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:47.609197 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:47.609908 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:47.610522 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:47.610527 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:47.610530 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:47.610763 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:47.611962 (+ 1199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:49:47.611970 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:49:47.612194 (+ 224us) client_negotiation.cc:769] Sending connection context
0926 16:49:47.612493 (+ 299us) client_negotiation.cc:240] Negotiation successful
0926 16:49:47.613183 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":86,"thread_start_us":494,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.253@KRBTEST.COM" created.
I20260926 16:49:47.618018 630 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.253" port: 44489 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:49:47.619437 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:47.618872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:44489 (local address 127.0.0.1:35158)
0926 16:49:47.619107 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:47.619245 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260926 16:49:47.619766 580 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.238.253:44489: connect: Connection refused (error 111)
I20260926 16:49:47.621913 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:47.621331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:44489 (local address 127.0.0.1:35164)
0926 16:49:47.621559 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:47.621692 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 16:49:47.622305 630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.253:44489: Network error: Client connection negotiation failed: client connection to 127.9.238.253:44489: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.253.keytab.
Entry for principal kudu/127.9.238.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.253.keytab.
I20260926 16:49:47.657553 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.253:44489
--webserver_interface=127.9.238.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.253.keytab
--principal=kudu/127.9.238.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:43901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:32879,127.9.238.253:44489,127.9.238.252:33901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 16:49:47.671783 630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.253" port: 44489 } attempt: 1
I20260926 16:49:47.673466 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:47.672861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:44489 (local address 127.0.0.1:35166)
0926 16:49:47.673154 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:47.673324 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260926 16:49:47.674551 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:47.674208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:44489 (local address 127.0.0.1:35168)
0926 16:49:47.674346 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:47.674425 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 16:49:47.674744 630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.253:44489: Network error: Client connection negotiation failed: client connection to 127.9.238.253:44489: connect: Connection refused (error 111)
I20260926 16:49:47.743039 630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.253" port: 44489 } attempt: 2
I20260926 16:49:47.744625 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:47.744036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:44489 (local address 127.0.0.1:35182)
0926 16:49:47.744363 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:47.744483 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260926 16:49:47.745903 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:47.745548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:44489 (local address 127.0.0.1:35186)
0926 16:49:47.745704 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:47.745777 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 16:49:47.746109 630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.253:44489: Network error: Client connection negotiation failed: client connection to 127.9.238.253:44489: connect: Connection refused (error 111)
I20260926 16:49:47.837388 630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.253" port: 44489 } attempt: 3
I20260926 16:49:47.838516 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:47.838093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:44489 (local address 127.0.0.1:35198)
0926 16:49:47.838292 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:47.838387 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 16:49:47.839490 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:47.839153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:44489 (local address 127.0.0.1:35200)
0926 16:49:47.839294 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:47.839365 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 16:49:47.839695 630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.253:44489: Network error: Client connection negotiation failed: client connection to 127.9.238.253:44489: connect: Connection refused (error 111)
I20260926 16:49:47.983004 630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.253" port: 44489 } attempt: 4
I20260926 16:49:47.984794 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:47.984072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:44489 (local address 127.0.0.1:35204)
0926 16:49:47.984538 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:47.984651 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260}
I20260926 16:49:47.985867 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:47.985524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:44489 (local address 127.0.0.1:35212)
0926 16:49:47.985662 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:47.985757 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.253:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 16:49:47.986040 630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.253:44489: Network error: Client connection negotiation failed: client connection to 127.9.238.253:44489: connect: Connection refused (error 111)
W20260926 16:49:48.080117 637 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 16:49:48.080623 637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:49:48.080762 637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:49:48.080950 637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:49:48.090407 637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:49:48.090571 637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:49:48.090682 637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:49:48.090777 637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:49:48.090894 637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:49:48.103969 637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:43901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:32879,127.9.238.253:44489,127.9.238.252:33901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.253.keytab
--principal=kudu/127.9.238.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.253:44489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.9.238.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:49:48.105830 637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:49:48.107685 637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:49:48.122984 643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:48.123051 645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:48.122946 642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:49:48.124269 637 server_base.cc:1034] running on GCE node
I20260926 16:49:48.125300 637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:49:48.127038 637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:49:48.128357 637 hybrid_clock.cc:648] HybridClock initialized: now 1790441388128270 us; error 53 us; skew 500 ppm
I20260926 16:49:48.128836 637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:49:48 dist-test-slave-wt2j krb5kdc[550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441388, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:49:48.132750 637 init.cc:377] Logged in from keytab as kudu/127.9.238.253@KRBTEST.COM (short username kudu)
I20260926 16:49:48.134630 637 webserver.cc:460] Webserver started at http://127.9.238.253:38051/ using document root <none> and password file <none>
I20260926 16:49:48.135491 637 fs_manager.cc:359] Metadata directory not provided
I20260926 16:49:48.135681 637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:49:48.136059 637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:49:48.139504 637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-1/data/instance:
uuid: "f5e757c95c4b4b4a9e2f9c56d670696b"
format_stamp: "Formatted at 2026-09-26 16:49:48 on dist-test-slave-wt2j"
server_key: "749bcc10beab6cf98f0b89b16a6074db"
server_key_iv: "5f83f8fb2da29d6142371cbf51881b32"
server_key_version: "encryptionkey@0"
I20260926 16:49:48.140554 637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-1/wal/instance:
uuid: "f5e757c95c4b4b4a9e2f9c56d670696b"
format_stamp: "Formatted at 2026-09-26 16:49:48 on dist-test-slave-wt2j"
server_key: "749bcc10beab6cf98f0b89b16a6074db"
server_key_iv: "5f83f8fb2da29d6142371cbf51881b32"
server_key_version: "encryptionkey@0"
I20260926 16:49:48.146939 637 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260926 16:49:48.151192 652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:49:48.152971 637 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 16:49:48.153201 637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-1/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-1/wal
uuid: "f5e757c95c4b4b4a9e2f9c56d670696b"
format_stamp: "Formatted at 2026-09-26 16:49:48 on dist-test-slave-wt2j"
server_key: "749bcc10beab6cf98f0b89b16a6074db"
server_key_iv: "5f83f8fb2da29d6142371cbf51881b32"
server_key_version: "encryptionkey@0"
I20260926 16:49:48.153450 637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:49:48.173797 637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:49:48.178608 637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:49:48.178915 637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:49:48.200744 637 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.253:44489
I20260926 16:49:48.200799 703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.253:44489 every 8 connection(s)
I20260926 16:49:48.202389 637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-1/data/info.pb
I20260926 16:49:48.208063 704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:49:48.212738 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 637
I20260926 16:49:48.212989 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-1/wal/instance
I20260926 16:49:48.213690 10171 external_mini_cluster.cc:1428] Setting key 5eb1e63a948146d3a521a39b404a5ef1
I20260926 16:49:48.219477 704 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.254" port: 32879 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:49:48 dist-test-slave-wt2j krb5kdc[550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441388, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.253@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.9.238.252@KRBTEST.COM; defaulting to no policy
I20260926 16:49:48.248067 706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.224189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43166 (local address 127.9.238.254:32879)
0926 16:49:48.224978 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:48.224989 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:48.225969 (+ 980us) server_negotiation.cc:407] Connection header received
0926 16:49:48.228177 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:48.228186 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:48.228305 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:48.228436 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:48.230604 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.231885 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.234044 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.234802 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.234950 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:48.239537 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:48.239574 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:48.239581 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:48.239641 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:48.243825 (+ 4184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:48.245004 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:48.245010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:48.245013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:48.245183 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:48.245746 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:48.245752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:48.245756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:48.246166 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:48.246344 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:48.247541 (+ 1197us) server_negotiation.cc:299] Negotiation successful
0926 16:49:48.247792 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":353,"threads_started":1}
I20260926 16:49:48.249644 707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.224872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:32879 (local address 127.0.0.1:43166)
0926 16:49:48.225759 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:48.225812 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 16:49:48.227851 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:49:48.228814 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:49:48.228827 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:49:48.229579 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:49:48.230318 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:48.230341 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.232160 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.232166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:48.233792 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:48.233814 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.235116 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.235122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:48.235445 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:48.236455 (+ 1010us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:49:48.236483 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:49:48.239215 (+ 2732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:49:48.244067 (+ 4852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:48.244085 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:48.244107 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:48.244819 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:48.245357 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:48.245361 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:48.245364 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:48.245598 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:48.246478 (+ 880us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:49:48.246492 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:49:48.247029 (+ 537us) client_negotiation.cc:769] Sending connection context
0926 16:49:48.247414 (+ 385us) client_negotiation.cc:240] Negotiation successful
0926 16:49:48.248054 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":370,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.9.238.252@KRBTEST.COM" created.
I20260926 16:49:48.252295 704 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.253" port: 44489 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:49:48 dist-test-slave-wt2j krb5kdc[550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441388, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.253@KRBTEST.COM for kudu/127.9.238.253@KRBTEST.COM
I20260926 16:49:48.270691 707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.253447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:44489 (local address 127.0.0.1:35228)
0926 16:49:48.253753 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:48.253783 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 16:49:48.253957 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:49:48.254864 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:49:48.254871 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:49:48.255476 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:49:48.255764 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:48.255780 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.257339 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.257343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:48.258414 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:48.258427 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.259268 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.259273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:48.259489 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:48.260199 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:49:48.260217 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:49:48.263078 (+ 2861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:49:48.267874 (+ 4796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:48.267880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:48.267883 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:48.268403 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:48.268902 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:48.268906 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:48.268908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:48.269062 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:48.269818 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:49:48.269822 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:49:48.269984 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 16:49:48.270105 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 16:49:48.270363 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":3}
I20260926 16:49:48.271404 709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.253547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35228 (local address 127.9.238.253:44489)
0926 16:49:48.254366 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:48.254372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:48.254401 (+ 29us) server_negotiation.cc:407] Connection header received
0926 16:49:48.254459 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:48.254465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:48.254663 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:48.254796 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:48.255980 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.257156 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.258553 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.259121 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.259258 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:48.263389 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:48.263405 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:48.263427 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:48.263471 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:48.267641 (+ 4170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:48.268568 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:48.268580 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:48.268586 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:48.268754 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:48.269185 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:48.269191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:48.269197 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:48.269686 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:48.269902 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:48.270318 (+ 416us) server_negotiation.cc:299] Negotiation successful
0926 16:49:48.270631 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":636,"thread_start_us":464,"threads_started":1}
I20260926 16:49:48.274390 704 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:49:48.275844 707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.275225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:33901 (local address 127.0.0.1:58902)
0926 16:49:48.275460 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:48.275652 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260926 16:49:48.276122 656 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
I20260926 16:49:48.277947 707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.277564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:33901 (local address 127.0.0.1:58914)
0926 16:49:48.277731 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:48.277818 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 16:49:48.278177 704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:33901: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
I20260926 16:49:48.285367 630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.253" port: 44489 } attempt: 5
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.9.238.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.252.keytab.
Entry for principal kudu/127.9.238.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.252.keytab.
I20260926 16:49:48.292613 10171 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIzJvco/build/asan/bin/kudu
/tmp/dist-test-taskIzJvco/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.252:33901
--webserver_interface=127.9.238.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.252.keytab
--principal=kudu/127.9.238.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.9.238.212:43901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.9.238.254:32879,127.9.238.253:44489,127.9.238.252:33901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 16:49:48 dist-test-slave-wt2j krb5kdc[550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441387, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for kudu/127.9.238.253@KRBTEST.COM
I20260926 16:49:48.302117 709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.286337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35240 (local address 127.9.238.253:44489)
0926 16:49:48.286582 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:48.286590 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:48.286617 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:49:48.286719 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:48.286723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:48.286804 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:48.286933 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:48.288137 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.289120 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.290523 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.291323 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.291507 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:48.295683 (+ 4176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:48.295689 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:48.295693 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:48.295735 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:48.298967 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:48.299843 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:48.299851 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:48.299856 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:48.300000 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:48.300438 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:48.300448 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:48.300451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:48.300796 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:48.300978 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:48.301455 (+ 477us) server_negotiation.cc:299] Negotiation successful
0926 16:49:48.301715 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260926 16:49:48.302134 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.286196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:44489 (local address 127.0.0.1:35240)
0926 16:49:48.286398 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:48.286420 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 16:49:48.286532 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:49:48.287147 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:49:48.287151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:49:48.287717 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:49:48.287915 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:48.287924 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.289295 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.289299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:48.290291 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:48.290304 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.291600 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.291609 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:48.291890 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:48.292844 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:49:48.292881 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:49:48.295348 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:49:48.299156 (+ 3808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:48.299163 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:48.299168 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:48.299693 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:48.300128 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:48.300133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:48.300138 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:48.300301 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:48.300929 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:49:48.300935 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:49:48.301131 (+ 196us) client_negotiation.cc:769] Sending connection context
0926 16:49:48.301300 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 16:49:48.301701 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 16:49:48.303530 630 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } has no permanent_uuid. Determining permanent_uuid...
I20260926 16:49:48.305045 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.304356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:33901 (local address 127.0.0.1:58920)
0926 16:49:48.304654 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:48.304799 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 16:49:48.306470 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.306002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:33901 (local address 127.0.0.1:58922)
0926 16:49:48.306200 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:48.306320 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 16:49:48.306782 630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:33901: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
I20260926 16:49:48.327651 704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } attempt: 1
I20260926 16:49:48.329178 707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.328606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:33901 (local address 127.0.0.1:58936)
0926 16:49:48.328854 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:48.328987 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 16:49:48.330404 707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.329981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:33901 (local address 127.0.0.1:58946)
0926 16:49:48.330176 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:48.330279 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 16:49:48.330704 704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:33901: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
I20260926 16:49:48.358143 630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } attempt: 1
I20260926 16:49:48.359851 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.359187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:33901 (local address 127.0.0.1:58960)
0926 16:49:48.359450 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:48.359638 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 16:49:48.361030 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.360694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:33901 (local address 127.0.0.1:58966)
0926 16:49:48.360847 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:48.360930 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 16:49:48.361248 630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:33901: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
I20260926 16:49:48.399040 704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } attempt: 2
I20260926 16:49:48.400651 707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.400040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:33901 (local address 127.0.0.1:58970)
0926 16:49:48.400291 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:48.400452 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 16:49:48.401818 707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.401441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:33901 (local address 127.0.0.1:58986)
0926 16:49:48.401598 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:48.401686 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 16:49:48.402052 704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:33901: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
I20260926 16:49:48.429533 630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } attempt: 2
I20260926 16:49:48.431092 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.430501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:33901 (local address 127.0.0.1:58992)
0926 16:49:48.430797 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:48.430902 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 16:49:48.432389 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.432039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:33901 (local address 127.0.0.1:58996)
0926 16:49:48.432199 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:48.432288 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 16:49:48.432614 630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:33901: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
I20260926 16:49:48.493346 704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } attempt: 3
I20260926 16:49:48.494922 707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.494302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:33901 (local address 127.0.0.1:59008)
0926 16:49:48.494568 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:48.494705 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260926 16:49:48.496224 707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.495759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:33901 (local address 127.0.0.1:59014)
0926 16:49:48.496006 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:48.496097 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
W20260926 16:49:48.496476 704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:33901: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
I20260926 16:49:48.538861 630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } attempt: 3
I20260926 16:49:48.540423 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.539854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:33901 (local address 127.0.0.1:59028)
0926 16:49:48.540106 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:48.540209 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 16:49:48.541683 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.541254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:33901 (local address 127.0.0.1:59030)
0926 16:49:48.541481 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:48.541570 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260926 16:49:48.541922 630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:33901: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
I20260926 16:49:48.639722 704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } attempt: 4
I20260926 16:49:48.641288 707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.640665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:33901 (local address 127.0.0.1:59046)
0926 16:49:48.640973 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:48.641086 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260926 16:49:48.642457 707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.642090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:33901 (local address 127.0.0.1:59060)
0926 16:49:48.642246 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:48.642332 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 16:49:48.642799 704 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:33901: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
W20260926 16:49:48.654959 712 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 16:49:48.655296 712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 16:49:48.655431 712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 16:49:48.655596 712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 16:49:48.663725 712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 16:49:48.663826 712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 16:49:48.663914 712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 16:49:48.664018 712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 16:49:48.664126 712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 16:49:48.674342 712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.9.238.212:43901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.9.238.254:32879,127.9.238.253:44489,127.9.238.252:33901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/krb5kdc/kudu_127.9.238.252.keytab
--principal=kudu/127.9.238.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.9.238.252:33901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.9.238.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 26 Sep 2026 15:44:26 UTC on ccycloud.quasar-rjpcag.root.comops.site
build id 4674729
ASAN enabled
I20260926 16:49:48.675793 712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 16:49:48.677110 712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 16:49:48.689942 717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:48.690110 718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 16:49:48.690099 720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 16:49:48.691283 712 server_base.cc:1034] running on GCE node
I20260926 16:49:48.692143 712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 16:49:48.693681 712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 16:49:48.694931 712 hybrid_clock.cc:648] HybridClock initialized: now 1790441388694853 us; error 46 us; skew 500 ppm
I20260926 16:49:48.695309 712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 26 16:49:48 dist-test-slave-wt2j krb5kdc[550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441388, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 16:49:48.698988 712 init.cc:377] Logged in from keytab as kudu/127.9.238.252@KRBTEST.COM (short username kudu)
I20260926 16:49:48.701028 712 webserver.cc:460] Webserver started at http://127.9.238.252:35965/ using document root <none> and password file <none>
I20260926 16:49:48.701802 712 fs_manager.cc:359] Metadata directory not provided
I20260926 16:49:48.701900 712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 16:49:48.702239 712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 16:49:48.705375 712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-2/data/instance:
uuid: "85933246145d433892692d99315741d2"
format_stamp: "Formatted at 2026-09-26 16:49:48 on dist-test-slave-wt2j"
server_key: "0e7e5e783dca889082c1691a5b5179c3"
server_key_iv: "62cc63c93c4630db8e76524ce2b72ce2"
server_key_version: "encryptionkey@0"
I20260926 16:49:48.706200 712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-2/wal/instance:
uuid: "85933246145d433892692d99315741d2"
format_stamp: "Formatted at 2026-09-26 16:49:48 on dist-test-slave-wt2j"
server_key: "0e7e5e783dca889082c1691a5b5179c3"
server_key_iv: "62cc63c93c4630db8e76524ce2b72ce2"
server_key_version: "encryptionkey@0"
I20260926 16:49:48.712201 712 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 16:49:48.716336 727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 16:49:48.718076 712 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260926 16:49:48.718295 712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-2/data,/tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-2/wal
uuid: "85933246145d433892692d99315741d2"
format_stamp: "Formatted at 2026-09-26 16:49:48 on dist-test-slave-wt2j"
server_key: "0e7e5e783dca889082c1691a5b5179c3"
server_key_iv: "62cc63c93c4630db8e76524ce2b72ce2"
server_key_version: "encryptionkey@0"
I20260926 16:49:48.718530 712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 16:49:48.719166 630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } attempt: 4
I20260926 16:49:48.720687 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.720147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:33901 (local address 127.0.0.1:59072)
0926 16:49:48.720384 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:48.720497 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 16:49:48.721946 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.721599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:33901 (local address 127.0.0.1:59086)
0926 16:49:48.721759 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:48.721829 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 16:49:48.722173 630 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.9.238.252:33901: Network error: Client connection negotiation failed: client connection to 127.9.238.252:33901: connect: Connection refused (error 111)
I20260926 16:49:48.733634 712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 16:49:48.737824 712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 16:49:48.738178 712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 16:49:48.759389 712 rpc_server.cc:307] RPC server started. Bound to: 127.9.238.252:33901
I20260926 16:49:48.759491 778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.9.238.252:33901 every 8 connection(s)
I20260926 16:49:48.760923 712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-2/data/info.pb
I20260926 16:49:48.762758 10171 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIzJvco/build/asan/bin/kudu as pid 712
I20260926 16:49:48.763017 10171 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIzJvco/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790440988577176-10171-0/minicluster-data/master-2/wal/instance
I20260926 16:49:48.763674 10171 external_mini_cluster.cc:1428] Setting key 2454745217e0a2baa8eb4330717b53e9
I20260926 16:49:48.766950 779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 16:49:48.775728 779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.254" port: 32879 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:49:48 dist-test-slave-wt2j krb5kdc[550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.252@KRBTEST.COM
I20260926 16:49:48.788962 782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.765980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59102 (local address 127.9.238.252:33901)
0926 16:49:48.766936 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:48.766953 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:48.767003 (+ 50us) server_negotiation.cc:407] Connection header received
0926 16:49:48.768898 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:48.768923 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:48.769392 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:48.769805 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:48.771311 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.772803 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.774082 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.774787 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.775000 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:48.780226 (+ 5226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:48.780239 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:48.780260 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:48.780308 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:48.784106 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:48.785090 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:48.785099 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:48.785113 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:48.785332 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:48.785707 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:48.785712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:48.785716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:48.786419 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:48.786726 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:48.787191 (+ 465us) server_negotiation.cc:299] Negotiation successful
0926 16:49:48.787763 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":411,"threads_started":1}
Sep 26 16:49:48 dist-test-slave-wt2j krb5kdc[550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441388, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.252@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
I20260926 16:49:48.798142 784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.779325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.254:32879 (local address 127.0.0.1:43176)
0926 16:49:48.780416 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:48.780468 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 16:49:48.780719 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:49:48.781269 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:49:48.781279 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:49:48.782281 (+ 1002us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:49:48.782722 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:48.782736 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.783982 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.783988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:48.785313 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:48.785327 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.786522 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.786532 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:48.786811 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:48.787753 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:49:48.787794 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:49:48.790765 (+ 2971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:49:48.795237 (+ 4472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:48.795249 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:48.795254 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:48.795950 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:48.796428 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:48.796433 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:48.796437 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:48.796605 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:48.797268 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:49:48.797274 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:49:48.797441 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 16:49:48.797600 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 16:49:48.797857 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"thread_start_us":588,"threads_started":1}
I20260926 16:49:48.798503 783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.779230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43176 (local address 127.9.238.254:32879)
0926 16:49:48.780017 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:48.780026 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:48.780592 (+ 566us) server_negotiation.cc:407] Connection header received
0926 16:49:48.780892 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:48.780896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:48.781015 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:48.781144 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:48.782921 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.783804 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.785563 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.786236 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.786401 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:48.791040 (+ 4639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:48.791047 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:48.791051 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:48.791096 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:48.795031 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:48.796124 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:48.796128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:48.796131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:48.796303 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:48.796741 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:48.796745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:48.796747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:48.797124 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:48.797323 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:48.797789 (+ 466us) server_negotiation.cc:299] Negotiation successful
0926 16:49:48.798059 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":390,"threads_started":1}
I20260926 16:49:48.800208 779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.253" port: 44489 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:49:48 dist-test-slave-wt2j krb5kdc[550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.254@KRBTEST.COM
Sep 26 16:49:48 dist-test-slave-wt2j krb5kdc[550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441388, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.252@KRBTEST.COM for kudu/127.9.238.253@KRBTEST.COM
I20260926 16:49:48.811463 785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.792667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43186 (local address 127.9.238.254:32879)
0926 16:49:48.793790 (+ 1123us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:48.793795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:48.793818 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:49:48.793876 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:48.793879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:48.793949 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:48.794057 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:48.795858 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.797014 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.798771 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.799319 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.799464 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:48.804950 (+ 5486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:48.804957 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:48.804962 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:48.805007 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:48.808522 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:48.809360 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:48.809366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:48.809369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:48.809502 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:48.809920 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:48.809925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:48.809928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:48.810244 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:48.810550 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:48.810858 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 16:49:48.811085 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":968,"thread_start_us":734,"threads_started":1}
I20260926 16:49:48.817221 784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.801051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.253:44489 (local address 127.0.0.1:35246)
0926 16:49:48.801387 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:48.801410 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 16:49:48.801544 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:49:48.801921 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:49:48.801929 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:49:48.802434 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:49:48.802641 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:48.802655 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.804185 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.804189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:48.805268 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:48.805281 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.806073 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.806079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:48.806305 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:48.806896 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:49:48.806919 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:49:48.809826 (+ 2907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:49:48.814074 (+ 4248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:48.814080 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:48.814085 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:48.814627 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:48.815037 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:48.815042 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:48.815046 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:48.815360 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:48.816167 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:49:48.816172 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:49:48.816409 (+ 237us) client_negotiation.cc:769] Sending connection context
0926 16:49:48.816598 (+ 189us) client_negotiation.cc:240] Negotiation successful
0926 16:49:48.816919 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203}
I20260926 16:49:48.817487 709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.801194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35246 (local address 127.9.238.253:44489)
0926 16:49:48.801394 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:48.801399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:48.801474 (+ 75us) server_negotiation.cc:407] Connection header received
0926 16:49:48.801682 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:48.801687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:48.801761 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:48.801857 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:48.802833 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.804034 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.805435 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.805908 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.806014 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:48.810106 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:48.810114 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:48.810118 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:48.810174 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:48.813886 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:48.814748 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:48.814753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:48.814755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:48.814906 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:48.815537 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:48.815543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:48.815606 (+ 63us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:48.815995 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:48.816153 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:48.816719 (+ 566us) server_negotiation.cc:299] Negotiation successful
0926 16:49:48.817019 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 16:49:48.818715 779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 16:49:48 dist-test-slave-wt2j krb5kdc[550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.9.238.253@KRBTEST.COM
Sep 26 16:49:48 dist-test-slave-wt2j krb5kdc[550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441388, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.252@KRBTEST.COM for kudu/127.9.238.252@KRBTEST.COM
I20260926 16:49:48.836234 784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.819644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:33901 (local address 127.0.0.1:59114)
0926 16:49:48.819848 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:48.819866 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 16:49:48.819968 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:49:48.820307 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:49:48.820311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:49:48.820832 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:49:48.821139 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:48.821151 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.822286 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.822294 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:48.823274 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:48.823286 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.824085 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.824091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:48.824278 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:48.824940 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:49:48.824966 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:49:48.827671 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:49:48.832676 (+ 5005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:48.832683 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:48.832687 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:48.833391 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:48.833972 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:48.833978 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:48.833981 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:48.834209 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:48.835022 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:49:48.835027 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:49:48.835234 (+ 207us) client_negotiation.cc:769] Sending connection context
0926 16:49:48.835409 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 16:49:48.835791 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 16:49:48.836539 782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.819764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59114 (local address 127.9.238.252:33901)
0926 16:49:48.819956 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:48.819961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:48.819983 (+ 22us) server_negotiation.cc:407] Connection header received
0926 16:49:48.820068 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:48.820073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:48.820153 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:48.820274 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:48.821393 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.822138 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.823414 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.823910 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.824070 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:48.828000 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:48.828007 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:48.828011 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:48.828056 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:48.832414 (+ 4358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:48.833540 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:48.833546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:48.833551 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:48.833815 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:48.834352 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:48.834357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:48.834361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:48.834883 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:48.835045 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:48.835583 (+ 538us) server_negotiation.cc:299] Negotiation successful
0926 16:49:48.836043 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":131,"server-negotiator.queue_time_us":64}
I20260926 16:49:48.836567 786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.813531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35248 (local address 127.9.238.253:44489)
0926 16:49:48.814242 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:48.814247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:48.814267 (+ 20us) server_negotiation.cc:407] Connection header received
0926 16:49:48.814318 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:48.814322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:48.814383 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:48.814474 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:48.816590 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.818135 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.819919 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.820545 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.820684 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:48.826857 (+ 6173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:48.826868 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:48.826874 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:48.826937 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:48.832414 (+ 5477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:48.833545 (+ 1131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:48.833551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:48.833555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:48.833764 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:48.834468 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:48.834477 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:48.834482 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:48.834868 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:48.835193 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:48.835551 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 16:49:48.835944 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":385,"threads_started":1}
I20260926 16:49:48.842321 10171 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 16:49:48.843505 779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85933246145d433892692d99315741d2: Bootstrap starting.
I20260926 16:49:48.849025 779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85933246145d433892692d99315741d2: Neither blocks nor log segments found. Creating new log.
I20260926 16:49:48.845216 605 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b7d51ceaa460445cb2a43d70b5d415be"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43186
I20260926 16:49:48.850756 779 log.cc:824] T 00000000000000000000000000000000 P 85933246145d433892692d99315741d2: Log is configured to *not* fsync() on all Append() calls
I20260926 16:49:48.854967 779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85933246145d433892692d99315741d2: No bootstrap required, opened a new log
I20260926 16:49:48.866683 779 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85933246145d433892692d99315741d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d51ceaa460445cb2a43d70b5d415be" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 32879 } } peers { permanent_uuid: "f5e757c95c4b4b4a9e2f9c56d670696b" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 44489 } } peers { permanent_uuid: "85933246145d433892692d99315741d2" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } }
I20260926 16:49:48.867331 779 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85933246145d433892692d99315741d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85933246145d433892692d99315741d2, State: Initialized, Role: FOLLOWER
I20260926 16:49:48.868085 779 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85933246145d433892692d99315741d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d51ceaa460445cb2a43d70b5d415be" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 32879 } } peers { permanent_uuid: "f5e757c95c4b4b4a9e2f9c56d670696b" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 44489 } } peers { permanent_uuid: "85933246145d433892692d99315741d2" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } }
I20260926 16:49:48.869441 788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85933246145d433892692d99315741d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d51ceaa460445cb2a43d70b5d415be" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 32879 } } peers { permanent_uuid: "f5e757c95c4b4b4a9e2f9c56d670696b" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 44489 } } peers { permanent_uuid: "85933246145d433892692d99315741d2" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } } }
I20260926 16:49:48.869884 788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85933246145d433892692d99315741d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:49:48.871505 779 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85933246145d433892692d99315741d2 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 16:49:48.883203 799 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85933246145d433892692d99315741d2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:49:48.883427 799 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:49:48.881098 712 master_runner.cc:186] Error getting master registration for 127.9.238.254:32879: OK, instance_id { permanent_uuid: "b7d51ceaa460445cb2a43d70b5d415be" instance_seqno: 1790441387556969 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 16:49:48.891767 712 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 16:49:48.942092 704 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } attempt: 5
Sep 26 16:49:48 dist-test-slave-wt2j krb5kdc[550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441388, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.253@KRBTEST.COM for kudu/127.9.238.252@KRBTEST.COM
I20260926 16:49:48.958786 782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.943291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59116 (local address 127.9.238.252:33901)
0926 16:49:48.943535 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:48.943567 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:48.943594 (+ 27us) server_negotiation.cc:407] Connection header received
0926 16:49:48.943793 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:48.943799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:48.943880 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:48.943981 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:48.944770 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.945524 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.946700 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.947221 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.947360 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:48.950960 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:48.950971 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:48.950978 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:48.951042 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:48.955515 (+ 4473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:48.956509 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:48.956517 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:48.956523 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:48.956733 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:48.957092 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:48.957098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:48.957103 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:48.957622 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:48.957822 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:48.958169 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 16:49:48.958410 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 16:49:48.958824 707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:48.943134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:33901 (local address 127.0.0.1:59116)
0926 16:49:48.943398 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:48.943425 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 16:49:48.943616 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:49:48.944032 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:49:48.944036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:49:48.944454 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:49:48.944612 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:48.944620 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.945668 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.945675 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:48.946553 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:48.946563 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:48.947379 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:48.947385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:48.947638 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:48.948552 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:49:48.948574 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:49:48.950630 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:49:48.955760 (+ 5130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:48.955766 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:48.955771 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:48.956334 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:48.956834 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:48.956840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:48.956844 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:48.956979 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:48.957753 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:49:48.957760 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:49:48.957909 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 16:49:48.958069 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 16:49:48.958383 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260926 16:49:48.964030 704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5e757c95c4b4b4a9e2f9c56d670696b: Bootstrap starting.
I20260926 16:49:48.967219 704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5e757c95c4b4b4a9e2f9c56d670696b: Neither blocks nor log segments found. Creating new log.
I20260926 16:49:48.968585 704 log.cc:824] T 00000000000000000000000000000000 P f5e757c95c4b4b4a9e2f9c56d670696b: Log is configured to *not* fsync() on all Append() calls
I20260926 16:49:48.971479 704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5e757c95c4b4b4a9e2f9c56d670696b: No bootstrap required, opened a new log
I20260926 16:49:48.981967 704 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5e757c95c4b4b4a9e2f9c56d670696b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d51ceaa460445cb2a43d70b5d415be" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 32879 } } peers { permanent_uuid: "f5e757c95c4b4b4a9e2f9c56d670696b" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 44489 } } peers { permanent_uuid: "85933246145d433892692d99315741d2" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } }
I20260926 16:49:48.982409 704 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5e757c95c4b4b4a9e2f9c56d670696b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e757c95c4b4b4a9e2f9c56d670696b, State: Initialized, Role: FOLLOWER
I20260926 16:49:48.982883 704 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5e757c95c4b4b4a9e2f9c56d670696b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d51ceaa460445cb2a43d70b5d415be" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 32879 } } peers { permanent_uuid: "f5e757c95c4b4b4a9e2f9c56d670696b" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 44489 } } peers { permanent_uuid: "85933246145d433892692d99315741d2" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } }
I20260926 16:49:48.983928 802 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e757c95c4b4b4a9e2f9c56d670696b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d51ceaa460445cb2a43d70b5d415be" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 32879 } } peers { permanent_uuid: "f5e757c95c4b4b4a9e2f9c56d670696b" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 44489 } } peers { permanent_uuid: "85933246145d433892692d99315741d2" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } } }
I20260926 16:49:48.984352 802 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e757c95c4b4b4a9e2f9c56d670696b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:49:48.985131 704 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5e757c95c4b4b4a9e2f9c56d670696b [sys.catalog]: configured and running, proceeding with master startup.
W20260926 16:49:48.990695 733 tablet.cc:2369] T 00000000000000000000000000000000 P 85933246145d433892692d99315741d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 16:49:48.996498 813 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f5e757c95c4b4b4a9e2f9c56d670696b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 16:49:48.996681 813 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 16:49:48.999401 630 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } attempt: 5
Sep 26 16:49:49 dist-test-slave-wt2j krb5kdc[550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790441387, etypes {rep=17 tkt=17 ses=17}, kudu/127.9.238.254@KRBTEST.COM for kudu/127.9.238.252@KRBTEST.COM
I20260926 16:49:48.994630 637 master_runner.cc:186] Error getting master registration for 127.9.238.254:32879: OK, instance_id { permanent_uuid: "b7d51ceaa460445cb2a43d70b5d415be" instance_seqno: 1790441387556969 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 16:49:49.021524 637 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 16:49:49.022642 782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:49.000430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59132 (local address 127.9.238.252:33901)
0926 16:49:49.000619 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:49.000627 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:49.000662 (+ 35us) server_negotiation.cc:407] Connection header received
0926 16:49:49.000809 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:49.000815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:49.000906 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:49.001035 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:49.002514 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:49.003778 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:49.006204 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:49.007139 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:49.007316 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:49.013244 (+ 5928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:49.013255 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:49.013265 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:49.013326 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:49.018041 (+ 4715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:49.019518 (+ 1477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:49.019527 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:49.019532 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:49.019786 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:49.020511 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:49.020517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:49.020520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:49.020911 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:49.021043 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:49.021890 (+ 847us) server_negotiation.cc:299] Negotiation successful
0926 16:49:49.022231 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 16:49:49.023728 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:49.000147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.9.238.252:33901 (local address 127.0.0.1:59132)
0926 16:49:49.000437 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 16:49:49.000468 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 16:49:49.000653 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 16:49:49.001362 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 16:49:49.001370 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 16:49:49.002052 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0926 16:49:49.002328 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:49.002342 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:49.004000 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:49.004009 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:49.005915 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 16:49:49.005946 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 16:49:49.007372 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 16:49:49.007381 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 16:49:49.007763 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:49.008653 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 16:49:49.008688 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 16:49:49.012892 (+ 4204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 16:49:49.018299 (+ 5407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:49.018306 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:49.018311 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:49.019232 (+ 921us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:49.020015 (+ 783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 16:49:49.020042 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 16:49:49.020046 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 16:49:49.020321 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 16:49:49.021190 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 16:49:49.021195 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 16:49:49.021489 (+ 294us) client_negotiation.cc:769] Sending connection context
0926 16:49:49.022865 (+ 1376us) client_negotiation.cc:240] Negotiation successful
0926 16:49:49.023199 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 16:49:49.030326 630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7d51ceaa460445cb2a43d70b5d415be: Bootstrap starting.
I20260926 16:49:49.034701 630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7d51ceaa460445cb2a43d70b5d415be: Neither blocks nor log segments found. Creating new log.
I20260926 16:49:49.036118 630 log.cc:824] T 00000000000000000000000000000000 P b7d51ceaa460445cb2a43d70b5d415be: Log is configured to *not* fsync() on all Append() calls
I20260926 16:49:49.040481 630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7d51ceaa460445cb2a43d70b5d415be: No bootstrap required, opened a new log
I20260926 16:49:49.048386 630 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7d51ceaa460445cb2a43d70b5d415be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d51ceaa460445cb2a43d70b5d415be" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 32879 } } peers { permanent_uuid: "f5e757c95c4b4b4a9e2f9c56d670696b" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 44489 } } peers { permanent_uuid: "85933246145d433892692d99315741d2" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } }
I20260926 16:49:49.049046 630 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7d51ceaa460445cb2a43d70b5d415be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7d51ceaa460445cb2a43d70b5d415be, State: Initialized, Role: FOLLOWER
I20260926 16:49:49.049595 630 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7d51ceaa460445cb2a43d70b5d415be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d51ceaa460445cb2a43d70b5d415be" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 32879 } } peers { permanent_uuid: "f5e757c95c4b4b4a9e2f9c56d670696b" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 44489 } } peers { permanent_uuid: "85933246145d433892692d99315741d2" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } }
I20260926 16:49:49.051326 816 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7d51ceaa460445cb2a43d70b5d415be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d51ceaa460445cb2a43d70b5d415be" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 32879 } } peers { permanent_uuid: "f5e757c95c4b4b4a9e2f9c56d670696b" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 44489 } } peers { permanent_uuid: "85933246145d433892692d99315741d2" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } } }
I20260926 16:49:49.052307 630 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7d51ceaa460445cb2a43d70b5d415be [sys.catalog]: configured and running, proceeding with master startup.
I20260926 16:49:49.052409 605 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7d51ceaa460445cb2a43d70b5d415be [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 16:49:49.052914 605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7d51ceaa460445cb2a43d70b5d415be [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:49:49.057538 605 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7d51ceaa460445cb2a43d70b5d415be [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d51ceaa460445cb2a43d70b5d415be" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 32879 } } peers { permanent_uuid: "f5e757c95c4b4b4a9e2f9c56d670696b" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 44489 } } peers { permanent_uuid: "85933246145d433892692d99315741d2" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } }
I20260926 16:49:49.061272 679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b7d51ceaa460445cb2a43d70b5d415be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f5e757c95c4b4b4a9e2f9c56d670696b"
I20260926 16:49:49.061975 679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5e757c95c4b4b4a9e2f9c56d670696b [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:49:49.066407 679 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f5e757c95c4b4b4a9e2f9c56d670696b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7d51ceaa460445cb2a43d70b5d415be in term 1.
I20260926 16:49:49.067698 580 leader_election.cc:302] T 00000000000000000000000000000000 P b7d51ceaa460445cb2a43d70b5d415be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7d51ceaa460445cb2a43d70b5d415be, f5e757c95c4b4b4a9e2f9c56d670696b; no voters:
I20260926 16:49:49.072500 754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b7d51ceaa460445cb2a43d70b5d415be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "85933246145d433892692d99315741d2"
I20260926 16:49:49.073180 754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85933246145d433892692d99315741d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 16:49:49.077638 754 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 85933246145d433892692d99315741d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7d51ceaa460445cb2a43d70b5d415be in term 1.
I20260926 16:49:49.078895 605 leader_election.cc:288] T 00000000000000000000000000000000 P b7d51ceaa460445cb2a43d70b5d415be [CANDIDATE]: Term 1 election: Requested vote from peers f5e757c95c4b4b4a9e2f9c56d670696b (127.9.238.253:44489), 85933246145d433892692d99315741d2 (127.9.238.252:33901)
I20260926 16:49:49.091243 816 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7d51ceaa460445cb2a43d70b5d415be [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:49:49.092442 816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7d51ceaa460445cb2a43d70b5d415be [term 1 FOLLOWER]: Leader election won for term 1
I20260926 16:49:49.094159 816 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7d51ceaa460445cb2a43d70b5d415be [term 1 LEADER]: Becoming Leader. State: Replica: b7d51ceaa460445cb2a43d70b5d415be, State: Running, Role: LEADER
I20260926 16:49:49.095687 816 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7d51ceaa460445cb2a43d70b5d415be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d51ceaa460445cb2a43d70b5d415be" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 32879 } } peers { permanent_uuid: "f5e757c95c4b4b4a9e2f9c56d670696b" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 44489 } } peers { permanent_uuid: "85933246145d433892692d99315741d2" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } }
I20260926 16:49:49.110548 819 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7d51ceaa460445cb2a43d70b5d415be [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7d51ceaa460445cb2a43d70b5d415be. Latest consensus state: current_term: 1 leader_uuid: "b7d51ceaa460445cb2a43d70b5d415be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d51ceaa460445cb2a43d70b5d415be" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 32879 } } peers { permanent_uuid: "f5e757c95c4b4b4a9e2f9c56d670696b" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 44489 } } peers { permanent_uuid: "85933246145d433892692d99315741d2" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } } }
I20260926 16:49:49.111301 819 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7d51ceaa460445cb2a43d70b5d415be [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:49.118269 828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 16:49:49.144244 828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 16:49:49.177673 786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:49.122911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35250 (local address 127.9.238.253:44489)
0926 16:49:49.123181 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:49.123193 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:49.140910 (+ 17717us) server_negotiation.cc:407] Connection header received
0926 16:49:49.141213 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:49.141221 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:49.141359 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:49.141534 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:49.149592 (+ 8058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:49.150969 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:49.154415 (+ 3446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:49.155210 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:49.155321 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:49.162572 (+ 7251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:49.162581 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:49.162587 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:49.162643 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:49.167010 (+ 4367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:49.174067 (+ 7057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:49.174074 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:49.174079 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:49.174264 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:49.174791 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:49.174797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:49.174802 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:49.175304 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:49.175611 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:49.176963 (+ 1352us) server_negotiation.cc:299] Negotiation successful
0926 16:49:49.177262 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 16:49:49.180987 785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:49.122328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43188 (local address 127.9.238.254:32879)
0926 16:49:49.122639 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:49.122647 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:49.141208 (+ 18561us) server_negotiation.cc:407] Connection header received
0926 16:49:49.141686 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:49.141693 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:49.141791 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:49.141886 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:49.149937 (+ 8051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:49.151120 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:49.154050 (+ 2930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:49.154887 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:49.154995 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:49.168090 (+ 13095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:49.168101 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:49.168107 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:49.168168 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:49.172890 (+ 4722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:49.176139 (+ 3249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:49.176147 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:49.176151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:49.176313 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:49.177995 (+ 1682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:49.178001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:49.178005 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:49.178611 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:49.179014 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:49.180283 (+ 1269us) server_negotiation.cc:299] Negotiation successful
0926 16:49:49.180582 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 16:49:49.181852 754 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 85933246145d433892692d99315741d2 [term 1 FOLLOWER]: Refusing update from remote peer b7d51ceaa460445cb2a43d70b5d415be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260926 16:49:49.182226 658 tablet.cc:2369] T 00000000000000000000000000000000 P f5e757c95c4b4b4a9e2f9c56d670696b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 16:49:49.183924 819 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b7d51ceaa460445cb2a43d70b5d415be [LEADER]: Connected to new peer: Peer: permanent_uuid: "85933246145d433892692d99315741d2" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:49:49.186853 679 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f5e757c95c4b4b4a9e2f9c56d670696b [term 1 FOLLOWER]: Refusing update from remote peer b7d51ceaa460445cb2a43d70b5d415be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 16:49:49.187847 819 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b7d51ceaa460445cb2a43d70b5d415be [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5e757c95c4b4b4a9e2f9c56d670696b" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 44489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 16:49:49.190683 782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:49.141978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59136 (local address 127.9.238.252:33901)
0926 16:49:49.142215 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:49.142223 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:49.156464 (+ 14241us) server_negotiation.cc:407] Connection header received
0926 16:49:49.156791 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:49.156798 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:49.156918 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:49.157001 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:49.171770 (+ 14769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:49.173263 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:49.175478 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:49.176161 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:49.176272 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:49.182644 (+ 6372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 16:49:49.182650 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 16:49:49.182654 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 16:49:49.182697 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 16:49:49.186570 (+ 3873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:49.187467 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:49.187476 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:49.187482 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:49.187669 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 16:49:49.188159 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 16:49:49.188164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 16:49:49.188167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 16:49:49.188610 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 16:49:49.188846 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:49.189999 (+ 1153us) server_negotiation.cc:299] Negotiation successful
0926 16:49:49.190268 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 16:49:49.233404 788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85933246145d433892692d99315741d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7d51ceaa460445cb2a43d70b5d415be. Latest consensus state: current_term: 1 leader_uuid: "b7d51ceaa460445cb2a43d70b5d415be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d51ceaa460445cb2a43d70b5d415be" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 32879 } } peers { permanent_uuid: "f5e757c95c4b4b4a9e2f9c56d670696b" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 44489 } } peers { permanent_uuid: "85933246145d433892692d99315741d2" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } } }
I20260926 16:49:49.233889 788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85933246145d433892692d99315741d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:49:49.235353 802 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e757c95c4b4b4a9e2f9c56d670696b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7d51ceaa460445cb2a43d70b5d415be. Latest consensus state: current_term: 1 leader_uuid: "b7d51ceaa460445cb2a43d70b5d415be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d51ceaa460445cb2a43d70b5d415be" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 32879 } } peers { permanent_uuid: "f5e757c95c4b4b4a9e2f9c56d670696b" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 44489 } } peers { permanent_uuid: "85933246145d433892692d99315741d2" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } } }
I20260926 16:49:49.235759 802 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e757c95c4b4b4a9e2f9c56d670696b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:49:49.238399 816 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7d51ceaa460445cb2a43d70b5d415be [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b7d51ceaa460445cb2a43d70b5d415be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d51ceaa460445cb2a43d70b5d415be" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 32879 } } peers { permanent_uuid: "f5e757c95c4b4b4a9e2f9c56d670696b" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 44489 } } peers { permanent_uuid: "85933246145d433892692d99315741d2" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } } }
I20260926 16:49:49.238749 816 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7d51ceaa460445cb2a43d70b5d415be [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:49.242165 828 catalog_manager.cc:1347] Generated new cluster ID: 7ca2c2c92b2744ed8623a72a5e5eb048
I20260926 16:49:49.242334 828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 16:49:49.250334 819 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7d51ceaa460445cb2a43d70b5d415be [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b7d51ceaa460445cb2a43d70b5d415be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d51ceaa460445cb2a43d70b5d415be" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 32879 } } peers { permanent_uuid: "f5e757c95c4b4b4a9e2f9c56d670696b" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 44489 } } peers { permanent_uuid: "85933246145d433892692d99315741d2" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } } }
I20260926 16:49:49.250686 819 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7d51ceaa460445cb2a43d70b5d415be [sys.catalog]: This master's current role is: LEADER
I20260926 16:49:49.256515 842 mvcc.cc:205] Tried to move back new op lower bound from 7333647930070257664 to 7333647929774379008. Current Snapshot: MvccSnapshot[applied={T|T < 7333647930070257664}]
I20260926 16:49:49.261736 788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85933246145d433892692d99315741d2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b7d51ceaa460445cb2a43d70b5d415be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d51ceaa460445cb2a43d70b5d415be" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 32879 } } peers { permanent_uuid: "f5e757c95c4b4b4a9e2f9c56d670696b" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 44489 } } peers { permanent_uuid: "85933246145d433892692d99315741d2" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } } }
I20260926 16:49:49.262130 788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85933246145d433892692d99315741d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:49:49.276722 802 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5e757c95c4b4b4a9e2f9c56d670696b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b7d51ceaa460445cb2a43d70b5d415be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7d51ceaa460445cb2a43d70b5d415be" member_type: VOTER last_known_addr { host: "127.9.238.254" port: 32879 } } peers { permanent_uuid: "f5e757c95c4b4b4a9e2f9c56d670696b" member_type: VOTER last_known_addr { host: "127.9.238.253" port: 44489 } } peers { permanent_uuid: "85933246145d433892692d99315741d2" member_type: VOTER last_known_addr { host: "127.9.238.252" port: 33901 } } }
I20260926 16:49:49.277010 802 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5e757c95c4b4b4a9e2f9c56d670696b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 16:49:49.280545 828 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 16:49:49.290536 828 catalog_manager.cc:1504] Loading token signing keys...
I20260926 16:49:49.319449 828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7d51ceaa460445cb2a43d70b5d415be: Generated new TSK 0
I20260926 16:49:49.321020 828 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 16:49:49.457275 785 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:49:49.450467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43190 (local address 127.9.238.254:32879)
0926 16:49:49.450795 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:49.450804 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:49.455104 (+ 4300us) server_negotiation.cc:407] Connection header received
0926 16:49:49.455370 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:49.455377 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:49.455509 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:49.455638 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:49.456938 (+ 1300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43190: BlockingRecv error: recv got EOF from 127.0.0.1:43190 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260926 16:49:49.459460 786 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:49:49.442151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35264 (local address 127.9.238.253:44489)
0926 16:49:49.442472 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:49.442481 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:49.454254 (+ 11773us) server_negotiation.cc:407] Connection header received
0926 16:49:49.454490 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:49.454499 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:49.454637 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:49.454828 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:49.459115 (+ 4287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35264: BlockingRecv error: recv got EOF from 127.0.0.1:35264 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260926 16:49:49.461359 782 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:49:49.458517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59142 (local address 127.9.238.252:33901)
0926 16:49:49.458756 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:49.458762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:49.459439 (+ 677us) server_negotiation.cc:407] Connection header received
0926 16:49:49.459709 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:49.459715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:49.459802 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:49.459903 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:49.461061 (+ 1158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59142: BlockingRecv error: recv got EOF from 127.0.0.1:59142 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 16:49:49.502784 786 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:49:49.495865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35276 (local address 127.9.238.253:44489)
0926 16:49:49.496061 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:49.496067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:49.500365 (+ 4298us) server_negotiation.cc:407] Connection header received
0926 16:49:49.500798 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:49.500806 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:49.501002 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:49.501098 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:49.502475 (+ 1377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35276: BlockingRecv error: recv got EOF from 127.0.0.1:35276 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 16:49:49.547420 786 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:49:49.535292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35278 (local address 127.9.238.253:44489)
0926 16:49:49.535580 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:49.535587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:49.544836 (+ 9249us) server_negotiation.cc:407] Connection header received
0926 16:49:49.545558 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:49.545565 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:49.545687 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:49.545793 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:49.547064 (+ 1271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35278: BlockingRecv error: recv got EOF from 127.0.0.1:35278 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260926 16:49:49.557528 786 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:49:49.554899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35292 (local address 127.9.238.253:44489)
0926 16:49:49.555201 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:49.555212 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:49.555833 (+ 621us) server_negotiation.cc:407] Connection header received
0926 16:49:49.556088 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:49.556101 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:49.556191 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:49.556330 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:49.557202 (+ 872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35292: BlockingRecv error: recv got EOF from 127.0.0.1:35292 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260926 16:49:49.595587 786 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:49:49.592383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35308 (local address 127.9.238.253:44489)
0926 16:49:49.592742 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:49.592751 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:49.593577 (+ 826us) server_negotiation.cc:407] Connection header received
0926 16:49:49.593864 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:49.593870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:49.593989 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:49.594138 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:49.595268 (+ 1130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35308: BlockingRecv error: recv got EOF from 127.0.0.1:35308 (error 108)
Metrics: {"server-negotiator.queue_time_us":153}
W20260926 16:49:49.613036 786 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:49:49.610018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35316 (local address 127.9.238.253:44489)
0926 16:49:49.610278 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:49.610286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:49.611034 (+ 748us) server_negotiation.cc:407] Connection header received
0926 16:49:49.611303 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:49.611309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:49.611454 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:49.611653 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:49.612780 (+ 1127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35316: BlockingRecv error: recv got EOF from 127.0.0.1:35316 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
W20260926 16:49:49.637342 786 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:49:49.633882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35320 (local address 127.9.238.253:44489)
0926 16:49:49.634126 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:49.634132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:49.634809 (+ 677us) server_negotiation.cc:407] Connection header received
0926 16:49:49.635072 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:49.635080 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:49.635231 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:49.635396 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:49.636412 (+ 1016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35320: BlockingRecv error: recv got EOF from 127.0.0.1:35320 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260926 16:49:49.675068 786 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:49:49.672087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35334 (local address 127.9.238.253:44489)
0926 16:49:49.672319 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:49.672325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:49.673035 (+ 710us) server_negotiation.cc:407] Connection header received
0926 16:49:49.673388 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:49.673393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:49.673477 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:49.673583 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:49.674805 (+ 1222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35334: BlockingRecv error: recv got EOF from 127.0.0.1:35334 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260926 16:49:49.746738 786 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:49:49.744188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35342 (local address 127.9.238.253:44489)
0926 16:49:49.744373 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:49.744378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:49.744821 (+ 443us) server_negotiation.cc:407] Connection header received
0926 16:49:49.745133 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:49.745138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:49.745261 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:49.745378 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:49.746453 (+ 1075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35342: BlockingRecv error: recv got EOF from 127.0.0.1:35342 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 16:49:49.882148 786 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 16:49:49.879824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35350 (local address 127.9.238.253:44489)
0926 16:49:49.880116 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:49.880121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:49.880561 (+ 440us) server_negotiation.cc:407] Connection header received
0926 16:49:49.880792 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:49.880796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:49.880877 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:49.881017 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 16:49:49.881962 (+ 945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35350: BlockingRecv error: recv got EOF from 127.0.0.1:35350 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 16:49:49.886001 799 catalog_manager.cc:1259] Loaded cluster ID: 7ca2c2c92b2744ed8623a72a5e5eb048
I20260926 16:49:49.886135 799 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 85933246145d433892692d99315741d2: loading cluster ID for follower catalog manager: success
I20260926 16:49:49.889478 799 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 85933246145d433892692d99315741d2: acquiring CA information for follower catalog manager: success
I20260926 16:49:49.891827 799 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 85933246145d433892692d99315741d2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 16:49:49.999450 813 catalog_manager.cc:1259] Loaded cluster ID: 7ca2c2c92b2744ed8623a72a5e5eb048
I20260926 16:49:49.999647 813 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f5e757c95c4b4b4a9e2f9c56d670696b: loading cluster ID for follower catalog manager: success
I20260926 16:49:50.004400 813 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f5e757c95c4b4b4a9e2f9c56d670696b: acquiring CA information for follower catalog manager: success
I20260926 16:49:50.007900 813 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f5e757c95c4b4b4a9e2f9c56d670696b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 16:49:50.150094 786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:50.143382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35358 (local address 127.9.238.253:44489)
0926 16:49:50.143651 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:50.143656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:50.143931 (+ 275us) server_negotiation.cc:407] Connection header received
0926 16:49:50.144145 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:50.144149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:50.144320 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:50.144479 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:49:50.145097 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:50.145890 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:50.147620 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:50.148076 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:50.148202 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:50.148803 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:49:50.149406 (+ 603us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:49:50.149541 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:50.149760 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 16:49:50.149844 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 16:49:50.165485 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:50.157471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59156 (local address 127.9.238.252:33901)
0926 16:49:50.158347 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:50.158352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:50.158375 (+ 23us) server_negotiation.cc:407] Connection header received
0926 16:49:50.158600 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:50.158604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:50.158771 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:50.158961 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:49:50.159684 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:50.160444 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:50.162315 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:50.163031 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:50.163150 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:50.163864 (+ 714us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:49:50.164499 (+ 635us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:49:50.164625 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:50.164915 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 16:49:50.164999 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753,"thread_start_us":621,"threads_started":1}
I20260926 16:49:50.181209 786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:50.172769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35366 (local address 127.9.238.253:44489)
0926 16:49:50.172993 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:50.173002 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:50.174516 (+ 1514us) server_negotiation.cc:407] Connection header received
0926 16:49:50.174681 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:50.174689 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:50.174801 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:50.174894 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:49:50.175845 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:50.176720 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:50.178987 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:50.179510 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:50.179683 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:50.180400 (+ 717us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:49:50.180556 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:49:50.180668 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:50.180819 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 16:49:50.180915 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 16:49:50.182695 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:50.174019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59160 (local address 127.9.238.252:33901)
0926 16:49:50.174364 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:50.174375 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:50.175183 (+ 808us) server_negotiation.cc:407] Connection header received
0926 16:49:50.175459 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:50.175489 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:50.175649 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:50.175815 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:49:50.177194 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:50.178346 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:50.180446 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:50.181151 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:50.181293 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:50.181835 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:49:50.181984 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:49:50.182080 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:50.182259 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 16:49:50.182372 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260926 16:49:50.183866 918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 16:49:50.172690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43200 (local address 127.9.238.254:32879)
0926 16:49:50.175790 (+ 3100us) server_negotiation.cc:206] Beginning negotiation
0926 16:49:50.175796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 16:49:50.175815 (+ 19us) server_negotiation.cc:407] Connection header received
0926 16:49:50.175923 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 16:49:50.175927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 16:49:50.176080 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 16:49:50.176259 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 16:49:50.177002 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:50.177810 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:50.180063 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 16:49:50.180756 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 16:49:50.181683 (+ 927us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 16:49:50.181945 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 16:49:50.182936 (+ 991us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 16:49:50.183113 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 16:49:50.183412 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 16:49:50.183533 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2950,"thread_start_us":2820,"threads_started":1}
I20260926 16:49:50.189451 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 563
I20260926 16:49:50.191164 625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:49:50.307358 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 563
W20260926 16:49:50.335127 731 connection.cc:535] server connection from 127.0.0.1:59132 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 16:49:50.335959 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 637
I20260926 16:49:50.337298 699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:49:50.449913 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 637
I20260926 16:49:50.472141 10171 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskIzJvco/build/asan/bin/kudu pid 712
I20260926 16:49:50.473513 774 generic_service.cc:196] Checking for leaks (request via RPC)
I20260926 16:49:50.602097 10171 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIzJvco/build/asan/bin/kudu with pid 712
2026-09-26T16:49:50Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5023 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10217 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (401990 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST